Static Lists - Botnadzor Extension

Static lists are a core data structure in Botnadzor, providing:

Bot/spam account lists — VK IDs and domains of known bots
Insertion configurations — DOM modification configs for UI injection
Other reference data — Various lookup tables and metadata

The StaticListsService manages fetching lists from remote servers, caching them in IndexedDB, and combining remote data with local user additions/overrides.

Architecture

Key Features

Streaming ingestion — Lists are fetched as JSONL streams and processed line-by-line
Dual instances — Two IndexedDB tables per list (A/B swap for atomic updates)
Combining modes — Three modes: remoteOnly, localOnly, remoteWithLocalOverrides
Local overrides — Users can add items or override remote items
Pollable summaries — Reactive summary statistics (counts, timestamps)

StaticListsService Implementation

The service is registered in the background entrypoint:

src/entrypoints/background.ts

const staticListsService = new StaticListsService({
  aliasManagerForStaticApi,
  rootConfigService,
});

registerService(staticListsServiceKey, staticListsService);

// Populate lists when root config is ready
void populateInitialStaticListsIfNeeded({
  rootConfigService,
  staticListsService,
});

Constructor

The service initializes IndexedDB and creates pollable state for each list:

src/entrypoints/background/@services/static-lists-service.ts

constructor({
  aliasManagerForStaticApi,
  rootConfigService,
}: {
  aliasManagerForStaticApi: AliasManager;
  rootConfigService: RootConfigService;
}) {
  this.aliasManagerForStaticApi = aliasManagerForStaticApi;
  this.rootConfigService = rootConfigService;

  // Initialize IndexedDB with Dexie
  this.db = new Dexie("static-lists");
  this.db.version(2).stores({
    [metadataTableName]: "listId",
    ...Object.fromEntries(
      staticListDefinitionEntries.flatMap(([listId, listDefinition]) => [
        [
          generateRemoteTableName(listId, "a"),
          ["++", ...listDefinition.indexes].join(","),
        ],
        [
          generateRemoteTableName(listId, "b"),
          ["++", ...listDefinition.indexes].join(","),
        ],
        [
          generateLocalTableName(listId),
          ["++", ...listDefinition.indexes].join(","),
        ],
      ]),
    ),
  });

  // Create pollable state for each list
  for (const listId of staticListIds) {
    pollableListMetadataByListId[listId] = new Pollable<
      StaticListMetadata | undefined
    >(undefined);

    pollableListSummaryByListId[listId] = new Pollable<
      StaticListSummary | undefined
    >(undefined);
  }

  // Start syncing metadata with DB
  void this.startSyncingMetadataWithDb();
}

List Metadata

Each list has metadata stored in IndexedDB:

type StaticListMetadata = {
  listId: StaticListId;
  remoteActiveInstance: "a" | "b";
  remoteActive?: {
    startedAt: IsoDateTime;
    summary: unknown;
    updatedAt: IsoDateTime;
    upstreamInfo: {
      generatedAt: IsoDateTime;
      url: string;
    };
  };
  remoteNext?: {
    lockId: string;
    startedAt: IsoDateTime;
    summary: unknown;
    updatedAt: IsoDateTime;
  };
  combiningMode: "remoteOnly" | "localOnly" | "remoteWithLocalOverrides";
  localSummary?: unknown;
  localUpdatedAt?: IsoDateTime;
  combinedSummary?: unknown;
};

Dual Instance System

Each list has two IndexedDB tables: listId_remote_a and listId_remote_b.

Active instance — Currently serving reads
Next instance — Target for new data during updates

This allows atomic swaps: fetch new data into the inactive table, then switch remoteActiveInstance from a to b (or vice versa).

function generateRemoteTableName(
  listId: StaticListId,
  instance: StaticListRemoteInstance,
): string {
  return `${listId}_remote_${instance}`;
}

function pickAnotherInstance(
  instance: StaticListRemoteInstance,
): StaticListRemoteInstance {
  return instance === "a" ? "b" : "a";
}

Fetching Lists

Lists are fetched from the remote API as JSONL (JSON Lines) streams:

src/entrypoints/background/@services/static-lists-service.ts

public async populateListIfOutdated(
  listId: StaticListId,
  toleranceInMinutes: number | undefined,
): Promise<PopulateFromUrlIfOutdatedResult> {
  const listLogger = this.getListLogger(listId);
  listLogger.info("Populating from if outdated");
  const startedAt = Date.now();

  const lockId = nanoid(8);

  const rootConfig = await this.rootConfigService.get();
  const upstreamInfo =
    rootConfig.remoteSystemLookup.staticApi.listLookup[listId];

  try {
    const initialMetadata = await this.getListMetadata(listId);
    if (
      this.isListUpToDate(
        initialMetadata,
        upstreamInfo.generatedAt,
        toleranceInMinutes,
      )
    ) {
      listLogger.info("List is up to date");
      return { success: true, data: "updateNotNeeded" };
    }

    // Acquire lock
    const lockMetadata = await this.waitForAnotherLock(listId, initialMetadata);
    this.setListMetadata({
      ...lockMetadata,
      remoteNext: {
        lockId,
        startedAt: isoDateTimeSchema.parse(startedAt),
        summary: structuredClone(mutableSummary),
        updatedAt: isoDateTimeSchema.parse(startedAt),
        upstreamInfo,
      },
    });

    // Fetch JSONL stream
    const fetchResult = await fetchFromRemoteSystem({
      aliasManager: this.aliasManagerForStaticApi,
      urlSuffix: `/lists/${listId}.jsonl`,
    });

    if (!fetchResult.success) {
      return {
        success: false,
        error: `Failed to fetch list from static API (reason: ${fetchResult.reason})`,
      };
    }

    // Stream and parse JSONL
    const nextInstance = pickAnotherInstance(
      initialMetadata.remoteActiveInstance,
    );
    const nextTable = this.db.table<unknown>(
      generateRemoteTableName(listId, nextInstance)
    );
    await nextTable.clear();

    let storedItemCount = 0;
    const listDefinition = staticListDefinitionLookup[listId];
    const mutableSummary = listDefinition.createEmptySummary();

    for await (const line of streamLines(fetchResult.response.body)) {
      const receivedItemResult = listDefinition.receivedItemSchema.safeParse(
        JSON.parse(line),
      );

      if (!receivedItemResult.success) {
        listLogger.error("Invalid item received: {error}", {
          error: receivedItemResult.error.message,
        });
        continue;
      }

      const itemToStore = listDefinition.mapReceivedToStored(
        receivedItemResult.data,
      );

      itemsToStore.push(itemToStore);
      listDefinition.mutateSummary(mutableSummary, itemToStore);

      if (itemsToStore.length >= itemBatchSize) {
        await nextTable.bulkAdd(itemsToStore);
        itemsToStore = [];
        storedItemCount += itemsToStore.length;
      }
    }

    // Swap to new instance
    this.setListMetadata({
      ...metadataWithoutRemoteNext,
      remoteActiveInstance: nextInstance,
      remoteActive: {
        startedAt: isoDateTimeSchema.parse(startedAt),
        summary: finalSummary,
        updatedAt: isoDateTimeSchema.parse(startedAt),
        upstreamInfo,
      },
    });

    listLogger.info(
      "Populated {storedItemCount} items in {ms}ms",
      { storedItemCount, ms: Date.now() - startedAt },
    );

    return { success: true, data: "updated" };
  } catch (error) {
    listLogger.error("Unexpected error while populating: {error}", { error });
    return { success: false, error: String(error) };
  }
}

Streaming JSONL

Lists can be large (thousands of entries), so they’re streamed and processed in batches:

src/entrypoints/background/@services/static-lists-service.ts

async function* streamLines(
  readableStream: ReadableStream<Uint8Array>,
): AsyncGenerator<string> {
  const decoder = new TextDecoder();
  const reader = readableStream.getReader();
  let { value: chunk, done } = await reader.read();
  let buffer = "";
  while (!done) {
    buffer += decoder.decode(chunk, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop() ?? "";
    for (const line of lines) {
      yield line;
    }
    ({ value: chunk, done } = await reader.read());
  }
  buffer += decoder.decode();
  if (buffer) {
    for (const line of buffer.split("\n")) {
      if (line.length > 0) {
        yield line;
      }
    }
  }
}

List Summaries

Each list has a summary — aggregated statistics computed during ingestion:

type StaticListSummary = {
  count: number;
  // List-specific fields...
};

Summaries are maintained for:

Remote active — Summary of current remote data
Local — Summary of user additions
Combined — Combined summary (respects combining mode)

Summaries are pollable, so clients can reactively update UI when lists change:

let pollVersion: PollVersion | undefined;
while (true) {
  const result = await staticListsService.pollListSummary(
    pollVersion,
    "insertions",
  );
  pollVersion = result.version;
  
  console.log("Insertions count:", result.value.count);
}

Summary Computation

Summaries are computed incrementally during ingestion:

const mutableSummary = listDefinition.createEmptySummary();

for (const item of items) {
  listDefinition.mutateSummary(mutableSummary, item);
}

const finalSummary = structuredClone(mutableSummary);

Each list definition provides:

createEmptySummary() — Creates initial summary object
mutateSummary(summary, item) — Updates summary with new item
unmutateSummary(summary, item) — Reverses update (for overrides)

Combining Modes

The service supports three combining modes:

remoteOnly

Uses only remote data, ignores local additions/overrides.

if (combiningMode === "remoteOnly") {
  return findInTable(await this.getActiveRemoteTable(listId));
}

localOnly

Uses only local data, ignores remote data.

if (combiningMode === "localOnly") {
  return findInTable(this.getLocalTable(listId));
}

remoteWithLocalOverrides (default)

Combines remote and local data:

Start with all remote items
Override items matching local items (by first index)
Append pure-local items (not in remote)

// Get remote items
const remoteItems = await remoteTable.toArray();
const localItems = await localTable.toArray();

// Build local lookup by first index
const localByKey = new Map();
for (const item of localItems) {
  const key = item[firstIndex];
  localByKey.set(key, item);
}

// Apply local overrides
const result = remoteItems.map((remoteItem) => {
  const key = remoteItem[firstIndex];
  return localByKey.get(key) ?? remoteItem;
});

// Append pure-local items
const remoteKeys = new Set(remoteItems.map((item) => item[firstIndex]));
for (const [key, item] of localByKey) {
  if (!remoteKeys.has(key)) {
    result.push(item);
  }
}

return result;

Local Item Management

Users can add, update, or remove local items:

Add Local Item

await staticListsService.addLocalItem("bots", {
  vkId: 123456,
  reason: "Spam account",
});

Remove Local Item

await staticListsService.removeLocalItem(
  "bots",
  "vkId",  // index
  123456,  // value
);

Put Local Item (add or override)

await staticListsService.putLocalItem("bots", {
  vkId: 123456,
  reason: "Updated reason",
});

Summary Recomputation

After local changes, summaries are recomputed:

src/entrypoints/background/@services/static-lists-service.ts

private async recomputeLocalAndCombinedSummary(
  listId: StaticListId,
): Promise<void> {
  const listDefinition = staticListDefinitionLookup[listId];
  const localItems = await this.getLocalTable(listId).toArray();

  const mutableLocalSummary = listDefinition.createEmptySummary();
  for (const item of localItems) {
    listDefinition.mutateSummary(mutableLocalSummary, item);
  }

  const metadata = await this.getListMetadata(listId);
  const updatedMetadata = await this.recomputeCombinedSummaryForMetadata(
    listId,
    {
      ...metadata,
      localSummary: structuredClone(mutableLocalSummary),
      localUpdatedAt: isoDateTimeSchema.parse(Date.now()),
    },
  );
  this.setListMetadata(updatedMetadata);
}

Item Origin Tracking

The service tracks the origin of each item:

type StaticListItemOrigin = "remote" | "local" | "localOverride";

remote — Item only exists in remote list
local — Item only exists locally (pure addition)
localOverride — Local item overrides a remote item

const origin = await staticListsService.getItemOrigin(
  "bots",
  "vkId",
  123456,
);

if (origin === "localOverride") {
  console.log("This item overrides a remote entry");
}

Paginated Access

For large lists, the service provides paginated access:

const { items, totalCount } = await staticListsService.getItemsPage(
  "bots",
  { offset: 0, limit: 100 },
);

for (const { item, origin, valid } of items) {
  console.log(`Item:`, item, `Origin:`, origin, `Valid:`, valid);
}

console.log(`Total items: ${totalCount}`);

Search by Index

Lists have indexed fields for efficient search:

const { items } = await staticListsService.searchItems(
  "bots",
  { index: "vkId", value: 123456 },
);

if (items.length > 0) {
  console.log("Found bot:", items[0].item);
}

Update Orchestration

The background script orchestrates list updates:

src/entrypoints/background.ts

async function populateInitialStaticListsIfNeeded({
  rootConfigService,
  staticListsService,
}: {
  rootConfigService: RootConfigService;
  staticListsService: StaticListsService;
}) {
  let pollVersion: PollVersion | undefined;
  for (;;) {
    const result = await rootConfigService.poll(pollVersion);
    pollVersion = result.version;
    const rootConfig = result.value;

    if (isEqual(rootConfig, rootConfigSeed)) {
      logger.debug(
        "Root config is the same as the seed, skipping static lists population",
      );
      continue;
    }

    staticListsService.updateIfNeeded();
  }
}

This ensures lists are updated when:

Extension starts (if outdated)
Root config changes (new upstream list URLs)

Performance Characteristics

Fetch Performance

Streaming ingestion — Memory usage is constant regardless of list size
Batch writes — Items are written in batches of 1000 for optimal performance
IndexedDB — Provides fast indexed lookups

Read Performance

Indexed queries — O(log n) lookup by indexed fields
Full scans — O(n) for paginated access
Pollable state — Clients only re-fetch when data changes

Storage

Compression — JSONL format is compact
Dual instances — 2x storage for atomic updates
Local additions — Separate table, minimal overhead

Next Steps

Proxy Services

Learn how to access StaticListsService from content scripts

Insertion System

See how insertion configs are fetched and applied

Documentation Index

​Architecture

​Key Features

​StaticListsService Implementation

​Constructor

​List Metadata

​Dual Instance System

​Fetching Lists

​Streaming JSONL

​List Summaries

​Summary Computation

​Combining Modes

​remoteOnly

​localOnly

​remoteWithLocalOverrides (default)

​Local Item Management

​Add Local Item

​Remove Local Item

​Put Local Item (add or override)

​Summary Recomputation

​Item Origin Tracking

​Paginated Access

​Search by Index

​Update Orchestration

​Performance Characteristics

​Fetch Performance

​Read Performance

​Storage

​Next Steps

Proxy Services

Insertion System

Architecture

Key Features

StaticListsService Implementation

Constructor

List Metadata

Dual Instance System

Fetching Lists

Streaming JSONL

List Summaries

Summary Computation

Combining Modes

remoteOnly

localOnly

remoteWithLocalOverrides (default)

Local Item Management

Add Local Item

Remove Local Item

Put Local Item (add or override)

Summary Recomputation

Item Origin Tracking

Paginated Access

Search by Index

Update Orchestration

Performance Characteristics

Fetch Performance

Read Performance

Storage

Next Steps