crate-agent

Finds, downloads, screens and prepares 1968–1998 Middle Eastern source records (Turkey, Egypt, Lebanon, Iraq, Iran, Saudi Arabia) for chopping.

The middle link in the chain: crate-agent finds the material → sample-reader measures it → subverselab-sampler chops it.

Four commands

# 1a) Search Internet Archive for candidate records
python3 -m crate discover --out data/candidates.json

# 1b) Search YouTube for INSTRUMENTAL candidates (downloads nothing, returns a list)
python3 -m crate yt --instrumental-only

# 2) Download the Archive candidates
python3 -m crate fetch --country TR EG --limit 20

# 3) Measure and screen what was downloaded
python3 -m crate screen --top 25

The yt command does not download — it lists candidates and prints the download command; you run the download with your own yt-dlp.

discover can be narrowed by country (--country TR IR), fetch by artist (--artist "Umm Kulthum").

Why the era decision comes from the artist

The obvious approach was tried first: creator:("Umm Kulthum") AND year:[1968 TO 1998]. Across 14 artists it returned 8 records in total — and the "Bergen" query returned Edgar Bergen's English-language comedy records.

The cause is not a lack of material. This repertoire sits on Archive mostly as user uploads with the creator and year fields left empty: there are 116 Umm Kulthum and 194 Fairuz recordings and not one of them carries a year. The year filter was throwing away the entire Arabic and Persian catalogue.

So the era decision is taken from the artist, not the record: Umm Kulthum died in 1975, Abdel Halim Hafez in 1977 — there is no need to know the year of their recordings. Active periods live in data/seeds.json, one entry per artist.

Those dates are editorial — not measured, written from knowledge. Correct any line you think is wrong; the tool treats that file as the single source of truth. When a year can be found on the record it is used instead, and the era_basis field says which one applied (item_year:title or artist_active:1926-1973).

Filtering: negative, not positive

A one-word name can belong to somebody else. The "Bergen" search returned Edgar Bergen's 1937 American radio programme, and the audio screen could not catch it — being a genuinely old analogue recording, its analogue signature came out at 0.837. Decisions like that have to be made in the metadata.

The first fix was a positive requirement ("the title must contain 'lebanese'"). It destroyed recall:

artist	unfiltered	positive requirement	negative filter
Fairuz	55	2	40
Warda	39	3	39
Ebi	14	5	14
Hayedeh	76	22	75
Bergen (Edgar)	9	0	0

Because most legitimate uploads do not carry a country name in the title. The default path is therefore exclude — dropping known-irrelevant categories such as sermons, radio programmes and news broadcasts. require survives only for genuinely ambiguous names, and the artist's own name is never written there (if it is, the condition filters nothing).

What the screen measures

Hard limits: a candidate shorter than 45 seconds, with more than 2000 clips, more than a third silence, or a peak below −30 dBFS is not ranked.

What passes is ranked by analogue signature — the mean of the share of energy above 8 kHz, the stereo width and the noise floor. This is not a date estimate; there is no training set with known dates, so it is nothing more than a summary of three measurements.

It was seen making the distinction on a real batch:

analogue  >8kHz    width    floor    secs  file
   0.994  0.0003   0.000    -25.8    1071  Umm Kulthum - Fakarony
   0.918  0.0001   0.067    -29.4     637  Umm Kulthum - We Daret El Ayam
   0.572  0.0159   0.241    -16.4     405  Yıllar Utansın (Müslüm Gürses) Official Audio

Umm Kulthum's 18-minute mono transfer on top; Müslüm Gürses' modern remaster upload at the bottom (0.0159 above 8 kHz). When hunting a source record, that is exactly what is wanted.

No microtonality

Deliberate. Three methods were tried and none of them discriminated — the most promising gave a clean ranking on a small sample and collapsed at n=29 (arabesk 0.279 vs Turkish rap 0.272). Details and reasons are in memory: reference-microtonality-detection.

I do not score what I have not measured. Instead the repertoire is chosen by an artist list rather than inferred from the signal — a sharper filter anyway.

There is no instrument recognition (violin/ney/percussion) either, for the same reason: without labelled data I could not verify it.

The YouTube side: the problem is not the year, it is authenticity

What was missing on Archive was the year. On YouTube the material is plentiful but most results are not the original recording. A single search for "orhan gencebay instrumental" returned, in one list: original instrumentals from the official Topic channel, an AI cover, two human covers, a "Jazz & LoFi" rearrangement and a guitar solo. Audio measurement does not separate these — they are all clean recordings. The distinction is in the title and the channel.

Channel tier. X - Topic is YouTube's auto-generated, label-sourced channel; the cleanest audio is there (T3). The artist's own channel is T2, everything else T1.

The search term is in the country's language. Measured 2026-09-02:

region	term	result
TR	enstrümantal	works, Topic + official channels
EG/LB/SA/IQ	المقدمات الموسيقية	gold seam — the orchestral introduction before the song: section violins, ney, qanun, percussion, no vocal
IR	بی کلام	bad seam — returns karaoke and "new arrangement", not the original

On Umm Kulthum that term opened up compilations organised by composer: Baligh Hamdi's introductions (46 min), Riyad al-Sunbati's (36 min), a 30-song compilation (142 min).

Three traps, all three closed:

1. AI and human covers, LoFi/jazz remixes → exclude list
2. On the Arabic side عزف على الأورج (played on organ) and saxophone versions → later instrumental interpretations, not the original orchestra
3. A Topic channel hosts the whole album; if the title carries no instrumental marker the track has vocals ("Seni Buldumya") → even at T3, the title condition is enforced

Instrumental candidates do not compete in the same ranking as full recordings: a Googoosh clip with vocals and 20 million views would crush every instrumental, and that is not what the sampler wants.

Current state

data/candidates.json — 690 candidates from Internet Archive: TR 227, IR 210, EG 170, LB 51, SA 19, IQ 13.

data/youtube_candidates.json — 78 pure instrumental candidates from YouTube (T3 2, T2 15, T1 61). Examples: Orhan Gencebay originals from his Topic channel, the Rahbani Brothers' Intro I and II from Fairuz's official channel, Mohamed Mounir's numbered موسيقى فقط album tracks.

Tests

python3 tests/test_crate.py

29 tests, none of them need the network.

Depends on

• ../sample-reader — the measurement layer (imported by code path)
• ../subverselab-sampler — the chop layer

Other countries: same method, different markers

The "year in the title" trick that works in Turkish does not carry over to the other languages. Measured 2026-09-02:

channel	country	videos	record term	Gregorian year
آرشیو صفحه‌های ایران	IR	339	294	0
NayefMusic	EG/LB	400	207	1
Bader vinyl records	LB	87	82	51
مطرب المقام العراقي	IQ	400	13	0
saalhsh.	SA	64	16	0

But the Iranian archive did have years — in a different calendar and in Persian numerals: سالِ ۱۳۵۱ = 1351 Solar Hijri = 1972. labels.years_in now reads both (13xx + 621).

Name extraction: repetition counting, not language rules

Title layout differs in every language and writing a rule for each is brittle:

Turkish   Artist - Track            (Kervan's lyric videos reverse it)
Arabic    إنت عمري | ... - محمد عبدالوهاب     song first, then artist
Persian   ترانه‌ی سفر با صدای رامش             artist after "با صدای"

crate/segments.py writes no rules. It uses this instead: an artist's name repeats across a catalogue, a song title does not. Every segment of the title is extracted, counted across the channel, and the most repeated segment is taken as the artist. No knowledge of the language required.

Result: data/intl_finds.json — 97 unknown artists (IR 46, EG/LB 47, IQ 4, LB 3), 30 of them with year evidence. Channel list in data/label_channels_intl.json.