Gain fields (attention fields)¶
You have stimulus images on a pixel grid, and on every frame (a TR or a trial) some locations are special: attended, salient, rewarded, expected, where the target or a distractor is. The question is whether the stimulus near those locations drives visual cortex more or less than the same stimulus elsewhere. A gain-field PRF model answers it on the stimulus grid: a field around each special location multiplies the image before every voxel’s ordinary PRF pools it.
\(W_{t\ell k}\) (in the paradigm): weight of gain type \(k\) at location \(\ell\) on frame \(t\), usually 0 or 1.
\(\gamma_k\): one gain per gain type, signed (> 0 enhances, < 0 suppresses). Shared by all voxels.
\(A(g;\ell)\): the field shape centred on location \(\ell\), peak 1 (a Gaussian with a shared width
sigmaby default).\(\mathrm{RF}_v\): the voxel’s receptive field from the model underneath (
GaussianPRF2D,DifferenceOfGaussiansPRF2D, …), with the same amplitude convention.
With every gain at 0, the model predicts exactly what the PRF model underneath predicts. So you can fit the PRFs first and then ask how much the gain fields add.
Tutorial: Gain fields: attention modulates population receptive fields simulates a population, fits shared gains with the PRFs fixed, combines two fields and adds a field shape of your own (also Lesson 8: Gain fields: attention modulates PRFs).
Step 1: the paradigm¶
The paradigm holds the images and the per-frame weights.
braincoder.stimuli.make_gain_field_paradigm() (or model.make_paradigm)
builds it:
import numpy as np
from braincoder.models import GainFieldPRF2D
# Pixel grid: (n_grid, 2) x and y in degrees.
x = np.linspace(-6, 6, 41)
gx, gy = np.meshgrid(x, x)
grid = np.stack([gx.ravel(), gy.ravel()], 1).astype(np.float32)
# Every location a field can sit on: any number, anywhere.
angles = np.arange(8) * np.pi / 4
field_positions = 4 * np.stack([np.cos(angles), np.sin(angles)], 1) # 8 items at 4 deg
model = GainFieldPRF2D(grid, field_positions, gain_labels=['target', 'distractor'])
model.parameter_labels
# ['x', 'y', 'sd', 'baseline', 'amplitude',
# 'field_sigma', 'field_gain_target', 'field_gain_distractor']
# images: (n_trials, n_grid) or (n_trials, n_x, n_y), flattened like grid.
# One dict per trial: gain label -> location index, (x, y) tuple,
# list of locations, or {location: weight}.
assignments = [{'target': 2, 'distractor': 5},
{'target': 0}, # no distractor on this trial
...]
paradigm = model.make_paradigm(images, assignments, index=betas.index)
Columns: ('image', x, y) for every pixel, then
('field', gain_label, location_index). An array
W of shape (n_frames, n_locations, n_gains) works instead of the list
of dicts, which is handy for fractional weights (e.g. the fraction of a TR a
distractor was on). If the special things move freely, pass every position
that occurs as field_positions.
Step 2: fit¶
The field parameters (widths and gains) belong to the model, not to a voxel.
The parameters DataFrame repeats them on every row, and they must be equal
across rows. Fit them with shared_pars. The robust route is to fit the PRFs
first with the model underneath, then fit the shared terms with the PRFs held
fixed:
from braincoder.models import GaussianPRF2D
from braincoder.optimize import ParameterFitter
# (a) PRFs from the mapping data, as usual, with GaussianPRF2D.
prf_pars = ... # columns x, y, sd, baseline, amplitude
# (b) Gain fields with the PRFs fixed. Gains start at 0 (= the PRF model).
init = model.add_field_parameters(prf_pars, field_sigma=1.0)
fitter = ParameterFitter(model, betas, paradigm)
pars = fitter.fit(init_pars=init,
fixed_pars=model.prf_parameter_labels(),
shared_pars=model.field_parameter_labels(include_tied=False),
learning_rate=0.02, max_n_iterations=2000, r2_atol=1e-7)
pars.iloc[0][model.field_parameter_labels()]
Leave out fixed_pars to refine PRFs and gains jointly. Check recovery on
simulated data at your noise level (model.simulate(paradigm, pars,
noise=...)) before trusting a gain: gains and widths trade off when the
field is much wider or narrower than the items.
Time courses with an HRF¶
GainFieldPRF2DWithHRF (and GainFieldDoGPRF2DWithHRF) apply the
modulation to the images frame by frame, then the HRF. Several fields can
act at once. Below, a sustained field marks the location attended in each
block, with a zero-sum constraint (the gain at the attended location and the
gain at the 3 others sum to 0, so a uniform change of the whole display is
left to the PRFs), and a transient field, which shares its width, responds to
brief onsets. The transient field sits outside the clip at 0. A third term
scales the drive of the items against a mapping bar, since both are painted
as 1 in the images:
from braincoder.hrf import SPMHRFModel
from braincoder.models import GainFieldDoGPRF2DWithHRF, GainField
positions = 4 * np.array([[1, 1], [-1, 1], [-1, -1], [1, -1]]) / np.sqrt(2)
item_mask = np.hypot(grid[:, 0], grid[:, 1]) > item_radius # your geometry
model = GainFieldDoGPRF2DWithHRF(
grid, positions, ['attended', 'unattended', 'onset'],
hrf_model=SPMHRFModel(tr=1.5),
fields=[GainField('sustained', gain_labels=['attended', 'unattended']),
GainField('transient', gain_labels=['onset'], clamped=False)],
image_scales={'item_scale': item_mask},
ties={'sustained_gain_unattended': {'sustained_gain_attended': -1 / 3},
'transient_sigma': 'sustained_sigma'})
# W: (n_TRs, 4, 3). Column 0: 1 at the attended location of the block;
# column 1: 1 elsewhere; column 2: fraction of the TR an onset was on.
paradigm = model.make_paradigm(images, W, index=bold.index)
ties makes a parameter a function of others: a number (a constant, e.g.
0. to switch a gain off), another label (equal), or {label: coef} (a
linear combination). Tied parameters are filled in by the model, so
predictions and fitted values always satisfy the constraint, and you leave
them out of shared_pars (field_parameter_labels(include_tied=False)).
image_scales adds a positive model-level parameter per mask; its null
value is 1.
Extending¶
Several fields. fields is a list of GainField
(name, shape, gain labels, sign constraint, inside or outside the clip). The
parameters are named f'{name}_{shape_parameter}' and
f'{name}_gain_{label}'.
A new field shape is a subclass of FieldShape
with parameter labels, their transforms (unconstrained → native) and one
method, written with keras.ops so it runs on every backend:
from keras import ops
from braincoder.models import FieldShape, GainField, GainFieldPRF2D
class AnnulusField(FieldShape):
parameter_labels = ['radius', 'width']
transformations = ['softplus', 'softplus']
def profile(self, dx, dy, radius, width):
# dx, dy: (1, n_locations, n_grid); parameters: (n_batches, 1, 1)
r = ops.sqrt(dx ** 2 + dy ** 2 + 1e-12)
return ops.exp(-(r - radius) ** 2 / (2 * width ** 2))
model = GainFieldPRF2D(grid, positions, ['attended'],
fields=[GainField('ring', AnnulusField())])
# parameters ... 'ring_radius', 'ring_width', 'ring_gain_attended'
GaussianField and DoGField (centre minus a wider surround) ship with
braincoder.
Another combination rule. Pass combination=f with f(images,
field_sums) -> images (one field sum per field, in order), e.g. divisive:
lambda S, F: S / (1 + ops.softplus(sum(F))); or subclass and override
_combine.
Another PRF kernel. Combine GainFieldMixin
with any image PRF model:
from braincoder.models import GainFieldMixin, DivisiveNormalizationGaussianPRF2D
class GainFieldDNPRF2D(GainFieldMixin, DivisiveNormalizationGaussianPRF2D):
pass
Limitations¶
Field parameters are model-level (one value for all voxels in the parameters DataFrame). Fit different areas as separate models if their fields may differ.
Decoding (
get_stimulus_pdf) is not meaningful with these models: decode with the PRF model underneath.ParameterFitterhas no bounds; constrain gains withGainField(gains='positive' | 'negative')and widths withGaussianField(sigma_min=...).Memory: the modulated images are
n_frames x n_grid(shared fields never build a voxels x frames x pixels tensor), so the cost is close to the PRF model underneath.
API¶
- class braincoder.models.GainFieldPRF2D(grid_coordinates=None, field_positions=None, gain_labels=None, fields=None, combination='multiplicative', clamp=True, image_scales=None, ties=None, positive_image_values_only=True, verbosity=20, **kwargs)¶
Bases:
GainFieldMixin,GaussianPRF2DGaussian 2D PRF with gain fields. See
braincoder.models.gain_field.One line for the common case (one Gaussian field, signed gains):
model = GainFieldPRF2D(grid_coordinates, field_positions, ['attended', 'ignored']) model.parameter_labels # ['x', 'y', 'sd', 'baseline', 'amplitude', # 'field_sigma', 'field_gain_attended', 'field_gain_ignored']
- class braincoder.models.GainFieldMixin(grid_coordinates=None, field_positions=None, gain_labels=None, fields=None, combination='multiplicative', clamp=True, image_scales=None, ties=None, positive_image_values_only=True, verbosity=20, **kwargs)¶
Adds gain fields to any image PRF model (
GaussianPRF2Dand subclasses). Put it first in the bases:class GainFieldDNPRF2D(GainFieldMixin, DivisiveNormalizationGaussianPRF2D): pass
- Parameters:
grid_coordinates (array (n_grid, 2)) – Pixel coordinates, as for
GaussianPRF2D.field_positions (array (n_locations, 2)) – Every location a field can be centred on, in grid units. Any number, anywhere (they need not lie on the grid).
gain_labels (list of str) – Names of the gain types, e.g.
['attended', 'ignored']. The paradigm says per frame which location carries which label, with which weight.fields (list of GainField, optional) – Default: one Gaussian field named
'field'with signed gains for all labels.combination ('multiplicative' or callable) –
'multiplicative'(default):M = max(1 + sum(clamped fields), 0) + sum(unclamped fields) S_eff = S * M
(without
clampthemaxis dropped). A callable gets(images, field_sums), withfield_sumsone(n_batches, n_frames, n_grid)tensor per field infieldsorder, and returns the modulated images. Or override_combine().clamp (bool, default True) – Clip the clamped part of the modulation at 0, so that strong suppression cannot invert the stimulus.
image_scales (dict {name: mask}, optional) – Extra model-level parameters that scale part of the image before the fields act:
S <- S * (1 + sum_s (scale_s - 1) * mask_s).maskis(n_grid,)(bool or weights). Each adds a positive parametername(softplus); its null value is 1. Use it when two kinds of stimulus share one image but their relative drive is unknown (e.g. a mapping bar and task items).ties (dict {label: tie}, optional) – Parameters computed from others instead of being free.
tieis a number (constant), another label (equal to it), or{label: coef}(linear combination of native values). Examples: a zero-sum pair{'field_gain_b': {'field_gain_a': -1/3}}, two fields with one width{'b_sigma': 'a_sigma'}, a switched-off gain{'field_gain_c': 0.0}. Tied columns ignore their raw value (no gradient); their native value is filled in by the forward transform, so predictions and fitted parameters always satisfy the tie.**kwargs – Passed to the PRF model (e.g.
rf_normalization).
- add_field_parameters(prf_parameters, **values)¶
Append the field columns to per-voxel PRF parameters.
Gains default to 0 and image scales to 1 (the null model, which predicts exactly what the PRF model underneath predicts); shape parameters such as widths need a value:
pars = model.add_field_parameters(prf_pars, field_sigma=1.0)
- apply_ties(parameters)¶
Return a copy of a parameter DataFrame with the tied columns filled in.
- field_parameter_labels(include_tied=True)¶
Labels of the model-level parameters (fields, image scales): pass them as
shared_pars.
- get_modulated_images(paradigm, parameters)¶
The images as the PRFs see them,
S_t(g) * M_t(g): DataFrame frames x pixels.
- make_paradigm(images, assignments, index=None)¶
braincoder.stimuli.make_gain_field_paradigm()for this model.
- prf_parameter_labels()¶
Labels of the per-voxel PRF parameters.
- class braincoder.models.GainField(name='field', shape=None, gain_labels=None, gains='signed', clamped=True)¶
One field: a shape, the gain labels it responds to, its gain sign.
- Parameters:
name (str) – Prefix of its parameter labels (
f'{name}_sigma', …).shape (FieldShape, default GaussianField())
gain_labels (list of str or None) – Which of the model’s gain labels this field responds to (
None= all). Each gives a parameterf'{name}_gain_{label}'.gains ({'signed', 'positive', 'negative'}) – Constraint on the gains.
'signed'(default) leaves them unbounded, so the data decide between enhancement (> 0) and suppression (< 0). With'positive'/'negative'do not initialise a gain at exactly 0.clamped (bool, default True) – With the default combination rule and
clamp=True, a clamped field sits inside the clipmax(1 + F, 0); an unclamped one is added outside it (e.g. a brief transient on top of a sustained field:M = max(1 + F_sustained, 0) + F_transient).
- class braincoder.models.FieldShape¶
Spatial profile of a field, centred on a field location.
A new shape is a subclass with
parameter_labels: names of its parameters (e.g.['sigma']);transformations: one per parameter,'identity','softplus','sigmoid'or a(forward, inverse)tuple ofkeras.opsfunctions from unconstrained to native values;
Example: an annulus (ring of radius
radiusand widthwidth):class AnnulusField(FieldShape): parameter_labels = ['radius', 'width'] transformations = ['softplus', 'softplus'] def profile(self, dx, dy, radius, width): r = ops.sqrt(dx ** 2 + dy ** 2 + 1e-12) return ops.exp(-(r - radius) ** 2 / (2 * width ** 2))
- profile(dx, dy, **parameters)¶
Field value at offsets
(dx, dy)from the field location.dx, dyhave shape(1, n_locations, n_grid); every parameter has shape(n_batches, 1, 1). Return(n_batches, n_locations, n_grid). Peak 1 keeps gains interpretable as the modulation at the field centre.
- class braincoder.models.GaussianField(sigma_min=0.01)¶
Isotropic Gaussian, peak 1:
exp(-d^2 / (2 sigma^2)).sigma = sigma_min + softplus(raw), in grid units.
- class braincoder.models.DoGField(sigma_min=0.01)¶
Difference of Gaussians (“Mexican hat”): a centre of width
sigmaminussurround_weighttimes a surround of widthsurround_ratio * sigma. Peak1 - surround_weight.sigma = sigma_min + softplus,surround_ratio = 1 + softplus(surround wider than centre),surround_weight = sigmoid(0..1).
- braincoder.stimuli.make_gain_field_paradigm(images, assignments, field_positions, gain_labels, grid_coordinates=None, index=None, atol=0.0001)¶
Build the paradigm of a gain-field PRF model.
A gain-field model needs, for every frame (TR or trial), the stimulus image and which field locations carry which gain. This function puts both in one DataFrame.
- Parameters:
images (array (n_frames, n_grid) or (n_frames, n_x, n_y), or DataFrame) – Stimulus intensity on the model’s pixel grid, flattened in the same order as
grid_coordinates.assignments (array (n_frames, n_locations, n_gains), or list of dicts) – The weight W[t, l, k] of gain
kat locationlon framet(usually 0/1). As a list (one entry per frame), each entry maps a gain label to a location, a list of locations, or a{location: weight}dict. A location is an int index intofield_positionsor an(x, y)tuple that must match one of them. Missing labels andNonemean weight 0 (no modulation on that frame).field_positions (array (n_locations, 2)) – Every location a field can sit at, in grid units. If the attended or salient things move freely, pass all positions that occur.
gain_labels (list of str) – Names of the gain types, e.g.
['attended', 'ignored'].grid_coordinates (array (n_grid, 2), optional) – The model’s pixel coordinates, only used to label the image columns
('image', x, y); without it they are('image', pixel_index, nan).index (pandas Index, optional) – Row index (defaults to the index of
imagesif it is a DataFrame, elseframe= 0..n_frames-1). Give it the index of your data.
- Returns:
Columns
('image', x, y)for the pixels, then('field', gain_label, location_index).- Return type:
DataFrame (n_frames, n_grid + n_gains * n_locations)