# Phil Fluent vocabulary: sources and editorial method

This Lab contains **1,000 unique single-word forms in each of Indonesian, Spanish and European Portuguese**. Each card has an English study meaning, a topic, the original corpus rank and count, and a link to its Wiktionary entry. The downloadable data and the vocabulary data modules are distributed under **[Creative Commons Attribution–ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/)**.

## Frequency data

Frequency data is adapted from **Hermit Dave and FrequencyWords contributors**, [FrequencyWords](https://github.com/hermitdave/FrequencyWords), using the **2018 OpenSubtitles** word-frequency files. The upstream project identifies its content license as CC BY-SA 4.0 and its corpus as [OPUS OpenSubtitles 2018](https://opus.nlpl.eu/OpenSubtitles-v2018.php). The code's MIT license is separate from the data's license.

Original files retrieved 11 October 2026:

- [Indonesian `id_50k.txt`](https://raw.githubusercontent.com/hermitdave/FrequencyWords/master/content/2018/id/id_50k.txt)
- [Spanish `es_50k.txt`](https://raw.githubusercontent.com/hermitdave/FrequencyWords/master/content/2018/es/es_50k.txt)
- [Portuguese `pt_50k.txt`](https://raw.githubusercontent.com/hermitdave/FrequencyWords/master/content/2018/pt/pt_50k.txt)

The Portuguese collection uses the upstream **`pt`** file, distinct from the **`pt_br`** file. It includes Portugal vocabulary such as `tu`, `rapariga`, `miúdo`, `fixe`, `telemóvel`, `equipa`, `facto` and `connosco`. The source is a subtitle corpus, not a representative survey of all speech in Portugal; it does not prove dialect exclusivity.

The original 50,000-line frequency files are preserved in `sources/`. Their SHA-256 hashes are in `manifest.json`. `sourceRank` is the one-based line number in the **unfiltered original file**. `frequency` is its observed token count. Display order 1–1,000 describes the curated learning set; it must not be presented as the original unfiltered rank.

## English meanings

English senses and morphology are adapted from **Wiktionary contributors**, via **Tatu Ylonen's Kaikki.org / Wiktextract** extraction of the English-language Wiktionary. The downloaded dataset was extracted **3 October 2026**, from the **2 September 2026 English Wiktionary dump**.

- [Indonesian dictionary and download](https://kaikki.org/dictionary/Indonesian/index.html)
- [Spanish dictionary and download](https://kaikki.org/dictionary/Spanish/index.html)
- [Portuguese dictionary and download](https://kaikki.org/dictionary/Portuguese/index.html)
- [Wiktionary copyright and licensing](https://en.wiktionary.org/wiki/Wiktionary:Copyrights)
- [Wiktextract](https://github.com/tatuylonen/wiktextract)

Each vocabulary entry's `dictionaryUrl` links to the relevant Wiktionary word and language section; that page and its history identify the contributors. The selected dictionary gloss before adaptation is retained in each language's `*-selection.json` file. No quoted example sentences, photographs or audio were copied.

Some English meanings and notes reuse the app's existing original learning content. Additional AI-assisted editorial changes simplify dictionary prose, select everyday senses, resolve inflected forms, disambiguate high-frequency words and add usage notes. The adaptations, topic assignments and editorial vocabulary contributions in this dataset are included under CC BY-SA 4.0. The source authors do not endorse this app.

## Selection and changes

Selection proceeds in original frequency order. Personal names, subtitle formatting fragments, numbers, standalone abbreviations, unsupported dictionary entries, obsolete/rare senses and editorially unsuitable entries are excluded. Single lexical words with a hyphen, including Indonesian reduplication, can be included. Capitalization is preserved where appropriate (`Anda`). Some dated Portuguese spellings are excluded rather than silently assigned the modern spelling's rank. All exclusions before the final cutoff are recorded in `*-selection.json`.

The final source-rank cutoffs are **1,306 for Indonesian**, **1,029 for Spanish**, and **1,039 for Portuguese**, each yielding exactly 1,000 selected forms. A source word's count is shared by all of its senses; the data does not estimate how often the selected English meaning occurs.

Plural forms and conjugations remain distinct because the underlying source counts **word forms**, not lemmas. For example, an infinitive and its common conjugations may be separate cards. Spanish and Portuguese inflections have English person/tense cues where possible, with dictionary morphology in the note. Portuguese and Spanish third-person forms may also address a listener formally; English `he/she` is a concise cue, not an exhaustive account of pronoun use. A card's meaning is a learning aid, not a full dictionary entry.

The 12 topic groups are editorial navigation aids, not categories supplied by the frequency corpus. Words can naturally fit more than one topic. Counts per topic are allowed to vary rather than replacing frequent words to make equal-sized decks.

## Review and limits

The build checks exactly 1,000 entries per language, uniqueness after Unicode normalization and case folding, single-word targets, nonempty meanings, valid topics, increasing source ranks, exact corpus counts, and consistency between TypeScript and JSON. Common pronouns, function words, conjugations, polysemy and European Portuguese usage received additional AI-assisted editorial review. This is **not independent native-teacher certification** of every card.

Subtitles overrepresent dialogue, drama, conflict, informal expressions and some offensive words. Explicit usage notes identify common rude terms. The collection is therefore called **1,000 common words**, or **a curated frequency-based vocabulary set**, rather than a universal or exhaustive “top 1,000 lemmas”.

## Reproduce and reuse

`manifest.json` records the original download URLs, input checksums and final topic counts. The original frequency snapshots are supplied in `sources/`. To regenerate, download the three Kaikki `.jsonl.gz` files listed in the manifest to a cache as `id-complete.jsonl.gz`, `es-complete.jsonl.gz` and `pt-complete.jsonl.gz`; put the frequency snapshots there as `id-frequency.txt`, `es-frequency.txt` and `pt-frequency.txt`. Check their hashes against the manifest. Then run:

```text
python3 scripts/build-vocabulary-data.py --cache /path/to/cache
python3 scripts/validate-vocabulary-data.py
```

The generator reads `lib/vocabulary-data/editorial/*.tsv` and the existing app content, and emits the TypeScript, JSON, editorial override snapshot and selection records. Kaikki download URLs are periodically refreshed upstream, so a later download may not have the recorded hash; keep the matching input files to reproduce this exact revision. The compressed source files used for this revision were retained in the build's local cache.

When sharing or adapting this vocabulary data, retain the source attribution, indicate modifications, link the license, and share adapted vocabulary data under CC BY-SA 4.0 or a compatible license. This notice concerns the vocabulary dataset; it does not change unrelated application code licensing.