A semicolon-separated file whose decimal point was sniffed from a column
read_tracks() does not read for a role (a label 1.5), while its x, y,
time, angle or weight columns hold only values that could be digit grouping
(1.234), now stops with radiatR_ambiguous_decimal instead of loading
values that may be wrong by a factor of 1000. read_opts = list(decimal = ".")
loads it as before. Files whose role columns show a decimal point (a time
column 0.040) are unaffected. The same applies to EthoVision exports that
are not comma-separated, and to the other dialects that sniff the mark.
A ; file whose decimal sniff finds a point, and whose first data row's
quoted field runs past the first 20 lines, now needs the mark given in
read_opts.
read_tracks(read_opts = ) now reaches the built-in dialects, which used to
ignore it: every dialect takes decimal; the .read_any()-based ones
(trackmate, trex, sleap, idtrackerai_wide) also delim and ext
(text files) and sheet (Excel files); ethovision and anymaze delim.
An option a dialect cannot use or that does not fit the file's type (delim
or decimal for an Excel file, sheet for a text file),
or any option for a registered dialect without a read_opts argument, now
stops with radiatR_invalid_argument instead of being silently ignored.
read_opts entries are checked on every path: an unknown name (which $
used to match partially: decimals = "," was read as decimal) or a
decimal mark other than "." / "," stops, as does a name given twice. Every
refusal carries the offending entry as option. The decimal-mark hint on
malformed values now applies to dialects and to H:MM:SS durations, no
longer to Excel files, and the error carries decimal and
suggested_decimal.
read_tracks() without a dialect now refuses a read_opts entry that does
not fit the file's type, as dialects do: delim or decimal for an Excel or
Arrow file, sheet for a text or Arrow file (Arrow files were also let
through by the dialects). A folder of mixed text and Arrow files read with
read_tracks_dir(read_opts = list(delim = )) therefore stops. For a data
frame, read_opts takes only decimal: delim, sheet and ext stop
instead of being accepted without effect.
headings_frame() stops on an angle column held as text (character or
factor), with radiatR_malformed_values naming the values that are not
numbers, or radiatR_invalid_argument when all are. With
units = "radians" and the unit-circle convention a character column used
to load as a text heading column; otherwise these columns failed with bare
base-R errors ("non-numeric argument to binary operator", "'abs' not
meaningful for factors").
A bare date whose midnight does not exist in tz (a DST change at
midnight, as in Cairo or Sao Paulo) is the first instant of that day
(01:00 there) under every reader, and so is an Excel date cell at such a
midnight. readr and fread() gave 01:00 for a column of such dates and
read.table() an error; a date beside date-times stopped under readr. A
date that does not exist at all in tz is an error.
Two date-times that used to load now stop with an error naming them, as
bugs fixed: a leap second (23:59:60, or 235960 in ISO 8601 basic)
under readr, which rolled it over to the next minute, now stops with an
error saying leap seconds cannot be represented, under every reader; and a
date-time on a day tz skips altogether (2011-12-30 in Pacific/Apia)
under data.table::fread() and read.table(), which moved it a day back,
now stops with an error saying it does not exist in tz.
derive_headings(rule = "straight") measures turns between moving steps
only. A pause (a step of zero length) has no direction; it was read as a
turn of 0, so a pause hid any turn around it, padded runs, and a run that
began on a pause had heading 0 (East). Pauses now neither break a run nor
count toward min_len. On cpunctatus, 47 of 235 trials have no straight
run (20 before: 30 lost their run, 3 gained one), and 104 of the 185 with
a heading either way change by more than a degree (median 90 degrees
among those that change).
derive_headings(rule = "pca_axis", source = "velocity") takes the axis of
the steps about zero, not about their mean, so a track of equal steps has
their direction as its axis (it had none, or one set by rounding). Under
either source a track whose net displacement has no component along the
axis (one that ends where it started) has heading NA: the axis has no
sign, which had been left to the eigenvector routine. On cpunctatus,
source = "velocity" changes 198 of 235 headings by more than a degree
(median 6.4, max 86.4); source = "position" changes none; no new NA.
A built-in derive_headings() rule stops with a radiatR_invalid_argument
error naming any option in ... it does not read (e.g. rule = "distal", frame_select = "all", which only the body-part rules read), and lists the
options it takes. A misplaced option was silently ignored, and a longer
misspelling was taken by partial matching (min_length = 50 acted as
min_len); both now stop. return_coords is accepted by every built-in
rule. Registered and function rules still receive ... unchecked.
angle_convention, which no rule ever read (a heading is in the unit-circle
convention), is among the options that now stop: drop it from such calls.
The ethovision dialect now reads the text export EthoVision XT writes: a
first line giving the number of header lines, a block of trial details, a
names row and a units row, ;-separated (or , or tab), in latin-1, UTF-8
or UTF-16, with - for a value the program did not have. Neither it nor
anymaze read a real export before. A file is one track (id "1"); its
time is Trial time, in seconds; a row with the wrong number of fields
stops the load, naming the line, so a file cut off mid-row has to have its
partial last line removed. A table whose first line is its column header is
read as a data frame, no longer as a file.
ethovision: zone = is now bodypart = (a zone is an area of the arena
in EthoVision; the argument names body points). The default is the centre
point; bodypart = c("nose", "tail") gives their mean. zone = stops with
an error naming the new argument. The nose and tail columns are read by the
names EthoVision documents (X nose, Y tail), which no export available
to us contains.
The anymaze dialect now reads ANY-maze's Time, Centre position X,
Centre position Y export. Time is a duration (0:01:02.500), read as
seconds. The centre point alone is read: zone = is removed, as is
skip_units_row = (the dialect deleted the first row whenever its position
was not a number, which on a real export is the first sample). The columns
it used to look for (X Centre, Nose X Centre) are in no export found.
A file without a time column is an error, and so is a Time of plain
numbers, whose unit cannot be told: give a data frame with a difftime
column instead. Pass the file path, or read Time as text (e.g.
readr::read_csv(f, col_types = readr::cols(Time = "c"))): readr's own
time parsing drops the fraction of a second.
Both dialects follow the rule for missing points (see "Masked points" in
?read_tracks): a sample without a position has an NA position, is
counted in @meta$attrition, and stops the load under
on_invalid = "error" or is dropped under "drop". A malformed cell stops
the load (radiatR_malformed_values); both dialects coerced silently
before. Both declare their time as seconds. The mean of several EthoVision
points divided the sum of y by the count of x present; it now masks a point
missing either coordinate.
draw_tracks(), line_circle_intercept_df(),
line_circle_intercept_traj() and new_headings_frame() are no longer
exported. They are internal helpers: draw tracks with radiate() or
gg_traj(), intersect a segment with the unit circle with
line_circle_intercept(), and build a headings frame with
headings_frame().
read_tracks() reads a delimited file the same way whichever reader is
installed (see "Reading delimited files" in ?read_tracks), and stops
where the readers used to disagree. The fields of each line are counted
first, and a row with more or fewer fields than the header is a
radiatR_malformed_file error naming its line: fread() kept the rows
before it, read.table() wrapped a long row past the fifth line into
extra rows, and readr shifted the fields, with at most a warning; readr
dropped a last line cut off mid-row without one. So is a line of spaces
(readr skipped it; fread() stopped reading there), an unclosed quote,
a table written with row names or a separator at the end of every row
(one name fewer than fields), which each reader read differently, a first
line fread() skipped as a preamble, and a column name used twice in a
file read without a dialect (readr renamed both, so the file lost its id
column and became one track). Every row of the file must be read. Under
read.table() a # no longer cuts a cell short, an apostrophe no longer
swallows rows and an empty cell is missing (a blank id was a track of its
own); fread() reads past a blank line and keeps a text cell's
surrounding whitespace, as the other readers do. Column names lose their
surrounding whitespace under every reader (readr kept " id", so no id
was found). A gzip, bzip2 or xz file is read under fread() without
R.utils (it stopped asking for R.utils, or read an xz file as text). A
line break quoted in a column name or cell of a file with Windows (CRLF)
line endings is read as "\n" under every reader, as read.table() read
it (readr and fread() kept "\r\n", so mapping = list(time = "fr\name") found no column).
Ids that differ only by surrounding whitespace (" 1" and "1") stop
read_tracks() with a radiatR_ambiguous_ids error: readr and
read.table() read them as two tracks and fread() as one.
get_all_object_pos() reads its landmark and track files strictly: a cell
that is not a number (a decimal comma, a word) is a
radiatR_malformed_values error naming the file, row and column, where it
failed with non-numeric argument to binary operator; a line with fewer
than three fields is a radiatR_malformed_file error (fields past the
third are not read, and need not be on every line). An
odd number of landmark rows is a radiatR_malformed_file error naming the
file (it was a warning, then an error that named none). When no file holds
a valid trial it stops with radiatR_no_valid_trials instead of returning
NULL with a warning.
The deeplabcut dialect now reads the predictions CSV DeepLabCut writes:
header rows scorer / bodyparts / coords, and individuals for
multi-animal projects, which load as one track per individual
(dialect_args = list(individuals = ) selects some). The frame index is the
time, in frames. deeplabcut_multiheader is removed; read_tracks() names
deeplabcut as its replacement (radiatR_removed_dialect). A flat
<bodypart>_x table is still accepted as a data frame, no longer as a file.
Labelled training data (CollectedData_*.csv, indexed by image) is refused
with an error saying so; the bundled example is now
dlc_single_wasp_predictions.csv, a predictions file.
deeplabcut and sleap apply one rule to missing and low-confidence
points (see "Masked points" in ?read_tracks). A point is masked when its
coordinates are missing, its score is 0, or it is below likelihood_min /
score_min. A frame with a masked point has no position, so it stops the
load under on_invalid = "error" and is dropped under "drop";
dialect_args = list(partial = "mean") averages the points that remain
instead. Before, a threshold on one bodypart removed rows without saying so,
a threshold on several averaged whatever passed, and DeepLabCut's (0, 0)
placeholder for a frame with no detection was read as a position. The
per-bodypart columns are NA where the point is masked.
What those dialects masked is reported in one message (class
radiatR_attrition) and kept in Tracks metadata (@meta$attrition): per
file and point, the rows masked, the rows read and the rows left without a
position. c() row-binds the records of its inputs.
A threshold now needs a score column for every selected point, and must be
one number; a negative score is malformed (radiatR_malformed_values).
DeepLabCut and SLEAP columns are checked strictly, like the other dialects':
a threshold used to be compared as text when the score column was text
("1e-3" >= 0.9). bodypart must name distinct points, and two SLEAP
headers that differ only in case or punctuation are refused.
DeepLabCut and SLEAP time is declared as frames (files, and data frames
whose time column is a frame column), so fps alone gives elapsed seconds
and time_type = "seconds" is refused for them.
derive_headings(carry = ) now stops when a carried column has the name of
a column of the result (id, time, heading, ...). It used to replace
the derived values with the carried ones without a message, so a call that
relied on that now fails with radiatR_invalid_argument: rename the column
in the tracks first.
Tracks objects now enforce the invariants the rest of the package assumes
(see "Invariants" in ?tracks): no missing ids; finite, strictly increasing
time within each id; finite angles and coordinates; finite, non-negative
weights. Previously a missing id, a non-finite or repeated time, an invalid
weight, a non-finite relative coordinate (rel_x/rel_y), or any
non-finite coordinate written after construction was accepted, and a missing
time crashed the validator. tracks() now reports every such violation at
once as a radiatR_invalid_tracks error whose problems field lists the
rule, column, input row and id. read_tracks() passes that error on, after
its own checks for non-finite x/y (radiatR_invalid_rows) and for duplicate
keys across files. Consequences: relative-coordinate columns must be finite
when the object is built, so give them finite placeholders (or name the
roles without storing the columns) if set_reference() is to fill them;
radiate() on a data frame fails when group_col does not identify single
trajectories (it used to draw one path per group that hopped between
trials), so pass the trajectory id as group_col; and a Tracks saved by an
earlier version that breaks a rule fails at its next validObject() (any
transform), not when it is loaded.
load_tracks() and load_tracks2() have been removed. Despite the names
they loaded no tracks; they joined manifest columns onto the file table used
by the landmark pipeline. Pass the manifest to get_all_object_pos()
instead, via its new manifest and manifest_cols arguments:
get_all_object_pos(file_tbl = file_tbl, track_dir = track_dir, manifest = manifest).
manifest_cols replaces load_tracks2()'s colnames.
The cpunctatus_tracks dataset has been removed. It duplicated the table
already stored in cpunctatus; use as.data.frame(cpunctatus) (or
cpunctatus@data) for the same per-frame rows.
simulate_tracks(seed = ) no longer changes the session's random-number
state: the simulation is drawn under seed and the previous state is restored
on exit. Seeded output is unchanged. Code that relied on the call to seed
later draws (a bootstrap or Monte-Carlo test run afterwards) must now call
set.seed() itself.
read_tracks() stops on a malformed value instead of treating it as missing.
A cell in an x, y, angle, weight or time column that is present but does not
parse ("oops", "1,5", a stray header or units row) raises a
radiatR_malformed_values error, a subclass of radiatR_invalid_rows, that
names the column, the data rows, the offending values and the file (each
file's own rows in a multi-file import). on_invalid = "drop" does not apply
to it: "drop" removes only rows whose coordinates are missing or
non-finite. Blank cells and NA are still missing values, and nan and
inf are read as such. Previously one bad cell made read_tracks() report
every row of the column as non-finite, and "drop" then dropped them all; a
bad cell after the first 1000 rows of a file was read as missing when readr
is installed; and in a multi-file import the bad cell surfaced as a
type-clash error from vctrs.
Files are now read with column types guessed from every row and with no
digit-grouping mark, so 1,5 or 1,234.5 in a tab-separated file is
malformed rather than read as 15 or 1234.5. Pass
read_opts = list(decimal = ",") for decimal commas. Where the decimal mark
is a comma, one bad cell names that cell only, not every 1,5 in its
column, and a value holding a . is malformed in turn, since 1.234 could
as well be 1234. A semicolon-separated file is taken to use decimal commas
unless its first lines show decimal points and no decimal comma: a value
such as 1.5 or 0.125, which cannot be digit grouping. For a file
read_tracks() reads itself, read_opts = list(decimal = ) settles it
either way.
The check covers the columns read_tracks() converts itself and the time
column of every dialect but ANY-maze. An error raised inside a dialect does
not yet name the file or the source column, and counts rows after the
dialect's own filtering and reshaping. Coordinates that a dialect converts
itself (ANY-maze, EthoVision, DeepLabCut, SLEAP) are not yet checked this
way.
Text time columns are parsed strictly. Every value must match one timestamp
layout in full: YYYY-MM-DD HH:MM:SS, YYYY-MM-DD HH:MM or YYYY-MM-DD
(with - or /), or ISO 8601 YYYY-MM-DDTHH:MM:SS with an optional Z
for UTC; a bare date beside date-times is midnight. An ISO 8601 timestamp,
or one followed by stray text, used to be cut to midnight of its date
without warning; the first now keeps its time of day and the second is an
error. A text column of numbers is read as numbers (it was an error) and,
with fps, stored as frames. A factor time column is read by its labels,
not its level codes. A Date column becomes midnight timestamps instead of
day counts, and a difftime column is converted to seconds whatever its
units. time_type = "frames" with a date-time, date or duration column is
now a radiatR_time_type_mismatch error; it used to store epoch seconds as
frame numbers.
read_tracks() stops with a radiatR_missing_columns error when mapping
names a column that is not in the data. A missing angle column used to be
ignored without notice, and a missing x or y column gave an error that did
not name it. A regex_mapping pattern (see read_tracks_format()) that
matches no column is the same error, naming the role and the pattern.
The validate argument of read_tracks() is deprecated. It never skipped
anything, because Tracks invariants are checked whenever an object is
built; any value but TRUE now warns, and the argument will be removed.
The trackmate dialect stops with a radiatR_unlinked_rows error (a
radiatR_invalid_rows) when a file holds spots that belong to no track
(TRACK_ID None or blank, as in an "all spots" export). They used to be
pooled into one trajectory called None. Pass
dialect_args = list(unlinked = "drop") to leave them out, with a
radiatR_dropped_rows warning. Without a units row saying POSITION_T is
in seconds, the dialect now takes FRAME as the time column rather than
POSITION_T.
Built-in loader dialects are now kept only for tracking software that is
widely used and still developed. Eleven dialects are removed: tracktor,
boris_xy, dtrack, motchallenge, csv_bbox, wide_prefix_xy,
tum_traj, euroc_mav, gpx, geojson_linestring and nmea_gprmc.
read_tracks(dialect = ) with one of these names stops with a
radiatR_removed_dialect error. To keep reading such files, register your
own loader with register_loader_dialect(); a loader registered under a
removed name works as before. The bundled tracktor_example.csv is removed
with its dialect, and xml2 is no longer a suggested package.
dtrack_read() is removed. vignette("loaders") shows a dtrack loader of
a dozen lines, registered with register_loader_dialect(), that reads the
bundled millipede track files.
A dialect is now the only reader of its input. read_tracks(path, dialect = )
used to parse the file with the generic reader first and discard the result,
which printed repaired-name messages for headerless files and could fail
before the dialect ran. With several paths the dialect is called once per
file; a custom dialect that expected the whole vector of paths must be
changed to take one path. read_tracks("<dialect name>"), which passed the
name itself to the dialect as input, is no longer accepted.
The raw example track files for cpunctatus now ship as one archive,
inst/extdata/tracks.tar.gz, instead of the 470-file directory
inst/extdata/tracks/, which cuts about 2 MB from the installed size.
system.file("extdata", "tracks", package = "radiatR") no longer finds
anything; unpack the archive first with
untar(system.file("extdata", "tracks.tar.gz", package = "radiatR"), exdir = tempdir()) and use file.path(tempdir(), "tracks"). The files
themselves are unchanged.
derive_headings() rules velocity_mean, vm_fit, velocity_axis and
origin_mean leave out what has no direction: a step of zero length (a
pause) and a position at the origin. Each was counted as pointing along the
x axis (atan2(0, 0) is 0), which pulled the heading of any track with
pauses towards it. In cpunctatus a quarter of the steps are pauses, and
the default velocity_mean heading moves by more than 1 degree in 194 of
235 tracks (median shift 12 degrees). A track that never moves now has
heading NA; it had heading 0.
The weighting options of those rules now do something. weight_by (for
velocity_mean and vm_fit) and r_power (for origin_mean) were passed
to circular::mean.circular(), which has no weights argument, so every
value gave the unweighted mean. weight_by = "step_length" now weights each
step by its length, which is the direction from the first position to the
last, as net gives; r_power = 1 gives the bearing of the centroid.
weight_by defaults to "uniform" for velocity_mean and vm_fit (the
argument default was "step_length", which had no effect), so their
headings remain the mean of the step directions. velocity_axis keeps
"step_length", which it already applied. vm_fit returns the same heading
as velocity_mean: its von Mises fit raised an error that was caught, and
the maximum-likelihood mean direction is the mean direction in any case.
origin_mean stops with radiatR_invalid_argument when r_power is not one
finite number; such a value was read as 0.
The other derive_headings() rules no longer read a displacement of zero
length as pointing along the x axis. net, straight, goal_bias,
maxspeed_window and window_net give NA for a track that never moves
(they gave 0), pca_axis gives NA when the positions, or the steps under
source = "velocity", have no spread (it gave pi/2), and distal gives NA
for a track that stays at the origin. net also gives NA for a track that
ends where it started, and straight for a straight run made of pauses only
(3 of 235 tracks in cpunctatus). All are reported by on_missing.
window_net gives NA for a window whose first and last positions are the
same (350 of 39631 windows in cpunctatus, which had heading 0), so
on_missing now warns about them.
get_tracked_object_pos() and get_all_object_pos() with
radius_criterion = "first_past" give a landmark (x0/y0, x1/y1 in
meta$trial_limits) of NA, with a warning naming the trial, when no point
of the trial reaches that radius. The farthest point was substituted, with
a warning only for the inner radius. A trial that is not mapped (it starts
too far from the centre) has NA landmarks and order/vid_ord in
trial_limits, not 0. get_all_object_pos() keeps the trial_limits rows
of a file none of whose trials is valid. The bundled cpunctatus follows
suit: trial con_19_1, which never reaches the inner radius, now has NA
landmarks in meta$trial_limits instead of its farthest point (0.059,
0.015) for both. Nothing else in the dataset changes.
readr no longer decides how a delimited file's date-times are read: like
data.table::fread(), it hands them back as the text in the file, and one
parser reads them for every reader. That parser now also reads ISO 8601
basic and hour-minute times, zoned or not (20240101T100000,
20240101T100000Z, 2024-01-01T1000+02:00, 2024-01-01T10:00). A time
column that mixes zoned (Z, +02:00) and unzoned date-times, or holds
hours alone (2024-01-01 10), a trailing decimal point, seconds on some
values only or a basic date and time split by a space (20240101 100000)
now stops under readr too, with an error naming the values; so does an
offset after a space (2024-01-01 10:00:00 +0200), which fread() read.
A kept column (keep =) whose every value is in one date-time layout (or
is a date) is read as date-times in tz under every reader, where
fread() and read.table() gave its text; any other kept column is kept
as read, the same under every reader (readr typed some, such as hours
alone). So a kept column can now stop the load under fread() and
read.table(), where it stayed text: one holding a time that does not
exist in tz (a DST gap), a leap second, or zoned and unzoned date-times
in one layout. Files whose unused date-time columns were typed differently (a CSV
beside an xlsx) bind again. A time column of date-times read with
time_type = "seconds" or "frames" now stops with
radiatR_malformed_values under every reader, as it did under
read.table(); readr gave epoch seconds (or, for frames, a
radiatR_time_type_mismatch error).
A file given to the wrong built-in dialect now stops instead of loading
wrong positions or ids. The sleap dialect needs a frame_idx column
unless time_col names the time: it read any <name>_x / <name>_y pair
as a node, so a ToxTrac file loaded its bounding box and a TRex file its
midline. The trex dialect refuses a track column (Track, TrackID,
TRACK_ID, trajectory_id) unless id_col names the id: it ignored one,
so a ToxTrac file's tracks became one id and a TrackMate file's spots one
track each. id_col and time_col of trex and trackmate are now
matched as the headers are ("Track" failed with an untyped error) and a
name that matches no column stops with radiatR_dialect_error, as do the
other refusals of trex, trackmate, toxtrac and idtrackerai_wide,
which were plain errors. These refusals, sleap's, and a misnamed
bodypart, id_col or time_col in any built-in dialect name the file,
which a multi-file read did not say. sleap matches id_col / time_col
as it matches its headers, too.
The app has a "Decimal mark" control (Auto-detect / Point / Comma) beside
Delimiter for text uploads. It applies to previews, trajectory and headings
loads and the downloaded scripts, and an error about values written with
the other mark says which setting to choose. A tab or comma file with
decimal commas, and a ; file whose decimals look like digit grouping,
can now be loaded in the app.
The sleap dialect accepts a node name as it is written in the file
(bodypart = "Tail Base").
A dialect can keep facts about the file it read with
attr(out, "source_meta") (see ?register_loader_dialect).
read_tracks() stores one record per file in Tracks@meta$source_meta,
with the file's name and path; c() concatenates them. ethovision keeps
the export's header block (header: trial, arena and subject names, start
time, ...) and the units of its columns (units) there.
The message about masked points offers partial = "mean" only when the
position was taken from more than one point.
The app detects EthoVision and ANY-maze uploads.
The app detects DeepLabCut files by their header rows, multi-animal files
included. A flat <bodypart>_likelihood table is still offered to the
deeplabcut dialect, which now refuses it by name; choose "Generic CSV" to
load it as plain columns. A dialect's refusal of a file is shown as the app's error headline.
The app can load a file that has rows with no position. When a load stops
on rows with a missing x/y, or on TrackMate spots that belong to no track,
the error offers to drop those rows and continue. Nothing is dropped unless
the user agrees, and the app asks again on every later load. The number of rows
dropped is shown above the results, and the downloaded scripts load the file
the same way (on_invalid = "drop",
dialect_args = list(unlinked = "drop")). This lets idtracker.ai files,
TrackMate "all spots" exports and DeepLabCut or SLEAP files with masked
frames load in the app, which refused them before. Malformed values and
duplicate id/time rows are still errors.
New vignette vignette("agent-guide"): the angle conventions, calibration
rules, failure behaviour and common pitfalls an AI agent (or anyone
scripting radiatR) needs. AGENTS.md at the repository root points to it.
The circ_summarise() help now states that mean_dir_deg is in the
display convention rather than a plain conversion of mean_dir, and the
read_tracks() help that keep adds to the mapped columns (it said
NULL keeps every column; unmapped columns are dropped).
The main workflow and simulation walkthroughs are now website-only articles
(vignettes/articles/) rather than installed vignettes, so
vignette("radiatR") and vignette("simulate") are no longer available
offline; read them at https://johnkirwan.github.io/radiatR/articles/.
Together with the dataset removal this brings the source tarball under
CRAN's 5 MB limit, and R CMD check now fails if it grows past 4.5 MB.
The hosted Shiny app has moved from shinyapps.io to Posit Connect Cloud
(git-backed deploy). The shinyapps.yaml GitHub Actions workflow is removed;
Connect Cloud rebuilds inst/app from the repo and installs dependencies from
the committed inst/app/manifest.json, regenerated by the new
connect-manifest workflow.
Reading several track files at once in read_tracks() no longer needs the
optional dplyr package (it is bound with vctrs::vec_rbind()). purrr is no
longer a dependency; vctrs, already required by tibble and ggplot2, is now
imported directly.
radiate() draws clipped tracks much faster: clipping paths to the unit
circle (clip_tracks = TRUE, the default) is now vectorised, taking a plot
of the cpunctatus example data from about 6 s to about 0.1 s. The drawn
geometry is unchanged. line_circle_intercept() is now vectorised over its
arguments.
Nineteen help pages in the circular-plotting and circular-boxplot groups now
have runnable examples: circ_display(), directedness_arrow(),
compute_circ_interval(), add_circ_interval(), add_heading_interval(),
compute_circ_mean(), add_circ_mean(), add_heading_arrow(),
add_stacked_headings(), line_circle_intercept(), gg_traj(),
add_angle_rose(), add_vonmises_density(), add_circular_kde(),
add_wrappedcauchy_density(), add_critical_r(), add_critical_v_line(),
circ_boxplot_stats() and add_circular_boxplot().
read_tracks(<data frame>, read_opts = list(decimal = ",")) reads a column
of "1,5" text as 1.5; the option was ignored, and the error about such
values now suggests it for a data frame too.
App: a headings file read with the wrong decimal mark no longer has its id column plotted as the angles. The angle picker only listed numeric columns, so an angle column read as text was missing and the first numeric column (often the id) was preselected. A text column of numbers in the other mark is now listed and preselected, with a note saying which Decimal mark to set, and Continue stops with that advice.
App: the script downloaded for a headings upload reads the file with the
delimiter and decimal mark the app read it with. With the delimiter control
on "auto" (the default) a ; file was read with read.csv(), which
stopped, and a ; file with decimal points was read with read.csv2(),
which left its numbers as text. A latin-1 file is read with
fileEncoding = "latin1".
read_tracks() reads a zip archive without readr or data.table installed.
The read.table() fallback was handed the archive itself and stopped with
a misleading "0 row(s) were read" error; it now reads the archive's first
member, as readr and fread() do, a latin-1 one included.
launch_app() no longer writes to the global environment. The app sourced
its helper files there (functions and constants such as build_plot_spec()
and APP_SEED), which CRAN policy forbids; they now live in the app's own
environment.
A latin-1 delimited file (as Windows programs write) reads as its text
under every reader: a byte such as µ in a column name or é in an id
stopped the load with an untyped "invalid UTF-8" error, and one in another
text column was kept as an invalid string. As for a tracker's
dialect file, one line that is not valid UTF-8 makes the whole file
latin-1, so in a file that mixes the two, its UTF-8 characters read as two
latin-1 ones each (µ as µ).
read_tracks(dialect = "deeplabcut"): a latin-1 body-part name (a header
re-saved by a Windows program) reads as its text. It made the file "not a
DeepLabCut output file".
App: a headings upload's group columns, and the generic preview's mapped
id column, keep their text where reading them as numbers would merge
labels (1.1 and 1.10), as read_tracks() already did for loads. The
downloaded script names such group columns in colClasses so they are read
as text too.
App: a file read by a tracker's dialect that the generic preview cannot
make a table of (a raw EthoVision export) is previewed as its first lines.
It showed one column of preamble text under readr and could not be
previewed under data.table::fread().
A time that does not exist in tz (the hour skipped when clocks go
forward) stops with an error saying so under every reader; under fread()
and read.table() it said only "not a date-time". (A skipped day and a
leap second: see Breaking changes.) A column of zoned and unzoned
date-times in one layout stops with an error saying the zones are mixed,
not that a valid value is no date-time. A bare date on a day tz skips
reports its own row. An Excel date cell on a day tz skips altogether
stops the load only when its column is used or kept, and reports its row.
A multi-file import whose time column holds bare dates in one file and
date-times in another (a CSV of dates beside a CSV or an xlsx of
date-times) binds under every reader, each date the first instant of its
day in tz. fread() stopped with a type clash, and under readr a date
on a day tz skips was reported as a missing time.
derive_headings() no longer calls headings "trials" for a rule that gives
several per trial (window_net, crossing): the on_missing message
counts missing headings and the trials they come from, and the app's
attrition banner does the same. missing_ids lists each trial once, and
the new n_trials attribute gives the number of trials.
read_tracks(id_collision = "namespace") gave files that share a name less
extension (n1.csv and n1.xlsx, or a/n1.csv and b/n1.csv) the same
prefix, so their ids still collided. Such files are now prefixed with their
full name, or the end of their path that tells them apart (a/n1.csv).
Ids written with surrounding spaces and no unpadded twin ( 1, 10) are
the same ids under every reader: readr kept them as the text " 1", where
data.table::fread() and read.table() read the number 1.
The separator sniff of read_tracks() counts fields as the readers do, so
a separator inside a quoted field no longer counts: a ; file with ; in
a quoted label is read as ;. A file whose header alone has a different
number of fields (a table written with row names) is named as such; it was
read as one column. A delimited file that still reads as a single column
stops with a radiatR_malformed_file error saying no separator was found,
instead of "Need either an angle column or both x and y columns".
read_tracks() no longer stops with "invalid UTF-8" in the separator sniff
when the first lines of a delimited file hold a byte that is not UTF-8 in a
data cell that is not an id (a latin-1 label): the separator and decimal mark
are sniffed from such lines too. A latin-1 byte in a column name or in an id
column still stops the read.
read_tracks() with a tz other than "UTC" put a timestamp that names
its zone (2024-01-01T10:00:00Z, ...+02:00) at the wrong instant when
the file was read by data.table::fread() (readr not installed): the
instant was written back as UTC wall-clock text and read again in tz,
moving it by the zone's offset (10:00Z read in New York as 15:00Z), and its
fraction of a second was dropped. Such columns are now read again as the
text in the file. Text time also reads a UTC offset (+02:00, -0130,
+02, at most 14 hours), so a file with offsets gives the same instants
under every reader; read.table() refused them before. A Z or offset
is read after any accepted layout (2024-01-01 10:00:00Z too). On the
text path a column that mixes zoned and unzoned date-times is malformed,
as one mixing Z with unmarked times was.
gg_traj() with facets = stopped when the plot was drawn: it asked
facet_wrap() for free scales, which coord_equal() does not allow. The
panels now share fixed scales, so trajectories on the unit circle are drawn
at the same scale in every panel.
Ids are kept as labels in more cases (see "Ids" in ?read_tracks). Two
files of one import whose ids read as one number, written differently
(1.1 in one file and 1.10 in the other), are two ids: they were
reported as colliding, and id_collision = "namespace" labelled both
1.1. An id column typed as a date or time keeps its text where two
labels are one instant (2024-01-01 00:00 and 2024-01-01 00:00:00 were
one track). An Excel file's ids are compared with a delimited file's as
the text of their cells (100000), and fread()'s large whole numbers
(integer64) as text too. Where the id types still disagree, in tables
that cannot be read again for their text (two tables from one dialect, a
Parquet file), the load stops with radiatR_id_type_mismatch; it
stopped with an internal Can't combine error.
derive_headings() and circ_summary_headings() now require rule.
Called without it, derive_headings() stopped with an internal
'length = 16' in coercion to 'logical(1)' error, and
circ_summary_headings() fell back to "crossing", which then stopped for
want of circ0 and circ1. Both now stop with an error that lists the
available rules. derive_headings() also refuses a rule that is not a
single name or a function (a vector, NA, a number), which previously gave
the same internal error.
read_tracks(tz = ) now applies to every timestamp without a time zone in
a delimited or Excel file that read_tracks() reads itself. A timestamp the
file reader typed (YYYY-MM-DD HH:MM:SS with readr or data.table installed,
or an Excel date-time cell) was read as UTC whatever tz was, while one
left as text (YYYY/MM/DD HH:MM:SS) was read in tz: the same wall-clock
time came out hours apart, between two files of one import or between two
machines with different readers installed. A timestamp marked Z is still
UTC, and a bare date beside such timestamps is midnight in tz. A
wall-clock time that does not exist in tz (the hour skipped when clocks go
forward) stops the load instead of being moved an hour on. tz does not
yet reach a file read by a dialect, or a Parquet or Feather file.
read_tracks(format = ) honours the format. It passed all of its own
defaults on to read_tracks_format(), where they overrode the format: the
format's mapping and regex_mapping were discarded and the columns
guessed from their names (a format mapping x to pos_x read a column named
x if the file had one), and its angle unit, time type, time zone and
frame rate gave way to the defaults, so degrees were read as radians. Now
only the arguments the caller supplies override the format, mapping role
by role, and read_opts is passed on. An argument given as NULL counts
as not supplied (fps = NULL or keep = NULL used to delete the format's
frame rate or kept columns), a dialect the caller names replaces the
format's, and load_manifest(format = ) follows the same rule: it passed
on all of its own defaults, so a degrees format was read as radians there
too. A format's dialect_args reach its dialect: they were handed to
read_tracks() as arguments of its own, which stopped with "unused
argument".
Ids that are written differently are no longer merged into one track. A
file reader types an id column as numbers when it can, so in any delimited
file ids 1.1 and 1.10, 1 and 1.0, 1e1 and 10 (and 01 and 1
without readr) were read as the same id, with no condition.
read_tracks() now keeps such a column as the text in the file (T and
TRUE, read as one logical value, likewise). The same holds for the mapped
id column and for any column named id, track, trajectory, animal or
subject. Ids that read as distinct numbers are numbers as before. Where
the delimited files of one import disagree on the type of the id column,
the ids of each file are the text in that file; the import used to stop
with a type error, or read TRUE beside numeric ids as 1.
A time that does not exist in tz (the hour skipped when clocks go
forward), in an Excel date-time cell or in a delimited file read with
readr, stops the load with a radiatR_malformed_values error naming the
file, column, row and value, when it is in a column read_tracks() uses:
the time column or one kept with keep = . It was read as NA, which
blanked a kept column (without a word from an Excel file, with readr's
"parsing issues" warning from a delimited one) and gave a time column an
error that named neither the value nor the cause. In a column that is not
used it is still NA.
circ_summarise() and circ_summary() on a Tracks object now return the
von Mises concentration kappa; it was always NA. The estimator was
looked up as circular::est.kappa, a function the circular package does not
have, so every caller silently got its fallback instead. kappa is now
computed directly as the maximum-likelihood estimate from the mean resultant
length (the value circular::mle.vonmises() gives), weighted where weights
are in use. Both report NA for fewer than three valid angles (as
circ_summarise() already did) and Inf when all angles coincide, where
the estimate is unbounded.
circ_summary_headings(), boot_kappa_ci() and the starting value in
concentration_regression() already used this estimate and are unchanged.
The trackmate, toxtrac, idtrackerai_wide and trex dialects now read
the files those programs write. Checked against real exports:
Tracking_RealSpace.txt (Time (sec), Track, Pos. X (mm))
and the headerless Tracking.txt (6 or 11 columns) load; both used to stop
with "required columns not found". The id is <arena>_<track>.trajectories.csv (seconds,x1,y1,...) loads; it used to
stop with "no x_ids entry the file does not have.frame twice use it as the time; they used to fall
back to the time column.
These dialects check their source columns strictly, so a malformed value is
reported by the header as written, the row of the file and the file name.Dialects now state the unit of their time column, so seconds are no longer
taken for frames when an fps is supplied (elapsed time and speeds were
divided by it). See "Time units from a dialect" in ?read_tracks. An fps
given for a time column in seconds is not used, with a message; an explicit
time_type that contradicts the dialect is a radiatR_time_type_mismatch
error.
App: tracking software is now recognised from the file's header for TrackMate, ToxTrac and idtracker.ai exports. The header was reduced before it was lower-cased, so no header containing a capital letter ever matched. A file whose time is in seconds no longer needs a frame rate for the kinematics.
Numeric time loaded as seconds is now treated as seconds. A Tracks read
with read_tracks(time_type = "seconds"), or from a duration (difftime)
column, was handled as a frame index: elapsed_seconds(), track_speed()
and the other kinematics stopped and asked for a frame rate, and once one
was set they divided the seconds by it. Such a Tracks now records
time_unit = "seconds" in its metadata and needs no frame rate; a frame
rate set on it is kept but not applied. c() refuses to combine Tracks
whose time units differ. Numeric time of unstated unit is still a frame
index, and the error that asks for a frame rate now mentions
time_type = "seconds".
read_tracks() and read_tracks_dir() now work with dialect and several
files. They used to fail with is.data.frame(df) is not TRUE. Each file is
read by the dialect and coerced on its own, so a malformed value is reported
against its own file and row; ids repeated across files follow
id_collision, and a dialect that returns no id column gets one id per
file. A failure inside the dialect is re-raised as a radiatR_dialect_error
naming the file, unless it is already a radiatR condition.
read_tracks() names the file(s) that do not exist instead of reporting
that x must be a data frame or path.
read_tracks() ignores a guessed id column that holds no values. It uses
the next id-like column if there is one, and otherwise reads the file as one
trajectory, as it does when there is no id column. SLEAP writes an empty
track column for single-animal exports, so sleap_example.csv now loads
without dialect = "sleap". An id column named in mapping, or one that is
only partly empty, is still an error.
x, y, angle and weight columns stored as text or as a factor load when every value is a number. Previously every row was reported as non-finite.
The app's error banner names the column and an example value when an upload has a malformed cell, instead of the generic "not valid track data".
read_tracks(dialect = "sleap") now reads an export whose track column is
empty (SLEAP's output for untracked, single-animal data) as one trajectory
with id "1". Previously such a file failed to load, or with a single row
loaded with a missing id.
An angle within floating-point error below zero (for example -1e-17, or a
position whose y is float noise below 0) no longer makes tracks() fail.
It wrapped to exactly 2*pi, outside [0, 2*pi); it now wraps to 0.
The script downloaded from the Shiny app now derives headings in the frame
the app used. In the app's default relative frame the script omitted
coords, so derive_headings() fell back to its "absolute" default and
the script's headings (and every statistic computed from them) differed from
those shown in the app. The frame is now always written explicitly, as the
one the app resolved: a dataset with no relative coordinates gets
coords = "absolute".
The Shiny app's method preview now follows the "Heading frame" control. It always illustrated the relative frame, so with the frame set to absolute (or with data that has no landmark frame) the preview showed a different frame from the one the Results derive headings in.
The Shiny app no longer calls set.seed() on the shared R process when it
computes Monte-Carlo uniformity p-values, so one session can no longer shift
another's random draws. The mean-direction bootstrap CI arc and the RANSAC
heading rule (in both the method preview and the results) are now drawn under
the same fixed seed, so they stay put across redraws, and the downloaded
script sets that seed to reproduce them.
circ_summary_headings() with the default group_by = "id" no longer errors
("numbers of columns of arguments do not match") when any trial produced no
heading; such trials are returned with n = 0 and NA statistics.
get_trial_limits() (and so get_all_object_pos()) no longer returns NA
manifest metadata for the second and later trials of a video when the file
table has fewer rows than the video has trials.
circ_summary() on weighted Tracks now weights mean_dir. It was the
unweighted mean beside a weighted resultant_R and kappa, because
circular::mean.circular() ignores a weights argument. Results change for
any Tracks with a weight column, or with w = . kappa is NA unless
three angles have a positive weight; zero-weight angles counted before.
circ_summary() stops with an error of class radiatR_nonestimable, naming
the id, when a group's weights are all zero. It returned resultant_R = NaN
without comment.
circ_summary(w = ) checks the column it is given: a name that is not a
column, or a column with missing, non-finite or negative values, is an error
of class radiatR_invalid_argument. A misspelt name was ignored and the
summary computed unweighted; rows with a missing weight were dropped.
circ_summarise(display = ) requires a circ_display() object and says so
(class radiatR_invalid_argument). display = NULL failed with "argument is
of length zero".
ctrax loader dialect and its optional R.matlab dependency have been
removed. .mat files are no longer supported; read_tracks(dialect = "ctrax")
now errors as an unknown dialect. Pre-computed orientation angles remain
supported through the general ellipse_axis heading rule on any column.read_tracks() now fails on non-finite coordinate rows by default, reporting
the affected row numbers and ids; pass on_invalid = "drop" to drop them with
a warning instead of the previous silent drop (breaking).c(Tracks, ...) now rejects trajectory ids that collide across the objects
being combined, and errors when the inputs carry conflicting frame_rate,
distance_scale, distance_unit, or normalize_xy metadata (including one
input having it set and another not). Previously these were silently merged,
discarding one object's calibration in favour of the other's, and could
interleave rows from two different trajectories that happened to share an
id. Per-id/per-run metadata (reference, trial_limits, sim_conditions,
oob_points, oob_trials) is now merged/summed across all combined inputs
instead of only the first object's being kept.read_tracks() and read_tracks_dir() now preserve trajectory identity when
importing multiple files. Previously, reading a character vector of paths with
no id column (the common case, relying on id_from_filename = TRUE) silently
collapsed every file into a single "(multiple)" trajectory whose rows were
then interleaved by time across files. Each file now yields its own trajectory
keyed by the file-name stem. A new id_collision = c("error", "namespace")
argument controls what happens when a real id value appears under more than one
file: "error" (the default) stops with a message naming the colliding id and
files, and "namespace" prefixes every id with its file stem
(file::id). Multi-file imports with no id column and id_from_filename = FALSE
now fail with a clear error instead of the silent collapse, and duplicate
(id, time) keys produced by combining files are reported rather than passed
through.read_tracks(..., time_type = "frames", fps = ) no longer converts frame
indices to seconds twice. The time column is now kept as a raw frame index
and elapsed_seconds() (and everything built on it: track_duration(),
track_speed(), velocity, and turning-rate) performs the frame -> seconds
division exactly once, on demand, using the stored frame_rate().
Previously the loader divided by fps immediately and then stored the same
fps as the frame rate, so downstream kinematics divided by fps a second
time -- e.g. an expected 60 units/s speed came out as 3600 units/s. Loading
with time_type = "seconds" no longer stores a spurious frame rate either.wrappedcauchy_fit()'s convergence column now matches its documentation.
circular::mle.wrappedcauchy() reports convergence as a logical, which the
previous code passed through as.integer(), yielding 1 for a converged
fit -- the opposite of the documented "0 = converged". The column is now
0 when the fit converged and 1 when it did not, and the (incorrect)
"optim return code" wording in the docs has been corrected.test_mean_directions(), test_concentration(),
test_distributions()) now signal a structured radiatR_nonestimable
condition (carrying method, group, n, and reason) for non-estimable inputs,
so callers can distinguish expected non-estimable cases from bugs.set_frame_rate(ts, 30) when no rate is set. Previously, uncalibrated numeric-frame data produced
plausible-looking speed and turning-rate results based on the unsupported
default.results_ok()) so a
failed or partial run cannot emit an apparently-valid result.test_gof() tests whether a sample of headings fits a wrapped Cauchy
distribution (Watson's U^2 on the probability-integral transform, with a
parametric-bootstrap p-value), closing the wrapped-Cauchy half of the
non-von-Mises goodness-of-fit gap. Jones-Pewsey GOF remains deferred.test_uniformity() gains test = "pycke", the Pycke (2010) omnibus
uniformity test — like Hermans-Rasson, powerful against multimodal and
asymmetric departures from uniformity, with a Monte-Carlo p-value.radiate() no longer labels every trajectory by its id when a plot has many
tracks. The automatic fallback to the id/group column now applies only when
that column has at most 12 distinct values, so many-track plots (e.g. the
235-trial cpunctatus) render cleanly instead of being buried under
overlapping trial-id labels. Pass label_col explicitly to force labelling
regardless of track count.radiate(clip_tracks = TRUE) (the default) now also clips beyond-circumference
overshoot in the absolute frame (coords = "absolute"), not just the
relative frame. The relative frame is a rotation of the absolute one, so
rho is identical in both; the previous relative-only restriction let
coords = "absolute" plots (e.g. in the package vignette) draw tracks
visibly outside the unit circle.get_all_object_pos() now reports out-of-bounds track points (radius > 1)
with a single aggregated message across the whole manifest, instead of one
message per file.test_mean_directions,
test_concentration, and test_distributions run automatically across the
selected Group-by column, showing whether groups differ in mean direction,
concentration, or overall distribution (not just whether each is
individually non-uniform).normalize_xy), which was previously pinned on with no way
to opt out. The transform fits each trajectory to the unit circle
independently and preserves path shape, not bearings from a fixed arena
origin; the control labels it as such rather than as a calibration. Downloaded
figure, statistics, and kinematics scripts now emit the normalize_xy value
the app actually loaded with, instead of silently falling back to the package
default.Tracks, with support for absolute and relative
coordinate frames and frame-rate / time metadata.radiate(), including axial overlays, faceting,
grouping, perimeter labelling and circular boxplots.inst/app), also deployed at
https://johnkirwan.shinyapps.io/radiatR/.