Harvey interview questions

import numpy as np

def top_k_cosine(q, docs, k=5, eps=1e-12):
    n = np.linalg.norm(docs, axis=1, keepdims=True)
    d = docs / np.maximum(n, eps)
    qn = q / max(np.linalg.norm(q), eps)
    scores = d @ qn
    idx = np.argsort(-scores)[:k]
    return idx, scores[idx]

Cosine is the dot product of unit-length vectors, so normalise once at indexing and any dot-product index ranks by cosine. Raw dot product also rewards vector length. Follow how the embedding model was trained: cosine for most sentence-embedding models, dot product when the model puts meaningful signal, such as popularity, in the norm. Guard against zero vectors.

import numpy as np

def chunk(text, size=200, overlap=40):
    w = text.split()
    step = size - overlap
    stop = max(len(w) - overlap, 1)
    return [" ".join(w[i:i + size])
            for i in range(0, stop, step)]

def build_index(chunks, embed):
    E = np.array([embed(c) for c in chunks], float)
    return E / np.linalg.norm(E, axis=1, keepdims=True)

def retrieve(q, E, embed, k=4):
    v = np.asarray(embed(q), float)
    s = E @ (v / np.linalg.norm(v))
    top = np.argsort(-s)[:k]
    return top, s[top]

def hit_rate(results, gold):
    hits = [bool(set(r) & g) for r, g in zip(results, gold)]
    return sum(hits) / len(hits)

Normalise embeddings so retrieval is a dot product. Evaluate on questions labelled with their relevant chunk ids: hit rate or recall at k, plus MRR. The prompt wraps chunks as quoted evidence, says nothing inside them is an instruction, and requires a chunk id per claim. Delimiters reduce injection without stopping it, so the model gets no tools. If the top score falls below a threshold tuned on the eval set, abstain.

Ingest memos with the metadata lawyers filter on, such as matter, client, jurisdiction and date, plus ethical-wall permissions enforced inside the retrieval query. Chunk along the memo's structure, issue, analysis, conclusion, keeping the heading path, so citations land on a paragraph. Use hybrid search, since case names and statute numbers need exact match, then a reranker.

The model answers only from retrieved passages with pinpoint citations, and a verifier checks each claim against its cited span. Flag memos the law may have overtaken, and abstain on conflicting evidence.

Split the work by what can be checked. A classifier labels the document type, and the type selects a rule set. Checkable rules run as code: required clauses present, dates valid, amounts consistent. The model handles judgment rules, and every finding cites the rule id, the exact span and a proposed edit. Suggestions arrive as a reviewable diff; humans accept, reject or override, and their decisions are logged as eval data.

Don't put every rule into one prompt. First map each rule to the clauses it touches. Then, per rule, a checker decides compliant or not and cites the span, and a reviser edits only failing clauses. Fixes can conflict, so re-check every rule against the revised contract and loop until all pass or a budget runs out, escalating conflicts to a lawyer. Output a tracked-changes diff and per-rule report.

Build environments that mirror the work: a sandbox with a document store, search and editing tools, and tasks spanning dozens of steps over long files. Warm-start with supervised fine-tuning on expert trajectories, then RL. Prefer verifiable rewards, like the right clauses found or the final document passing checks; use a calibrated judge for the rest, a small step penalty, and a curriculum from short to long tasks. Read transcripts for reward hacking.