Racket: Language-Oriented Programming Tutorial

Welcome to the world of Language‑Oriented Programming (LOP) with Racket! If you’ve ever felt constrained by a single programming language’s syntax and libraries, you’re about to discover how Racket lets you design tiny, purpose‑built languages that feel like natural extensions of your own ideas. In this tutorial we’ll explore the core concepts of LOP, walk through two practical language extensions, and see how they can simplify real‑world problems—from domain‑specific data pipelines to educational DSLs.

Why Racket is the Perfect Playground for LOP

Racket started life as a Scheme dialect, but it has grown into a full‑featured ecosystem that treats languages as first‑class citizens. Its macro system is powerful enough to rewrite syntax, and its module system lets you package languages just like any other library.

Because Racket itself is written in Racket, you can experiment with language design without leaving the comfort of a familiar environment. This tight feedback loop is what makes Racket the go‑to platform for researchers, educators, and hobbyists who want to prototype new syntactic ideas quickly.

Key Features that Enable LOP

• Macros: Transform code before it’s evaluated, enabling custom syntactic forms.
• #lang directive: Declare the language a file is written in, making language switching seamless.
• Reader extensions: Hook into the parsing phase to introduce new literal forms.
• Module system: Encapsulate language definitions and reuse them across projects.

Pro tip: Start every new language experiment in its own submodule. This keeps the global environment clean and makes debugging macros much easier.

Building Your First DSL: A Simple Query Language

Imagine you need to filter a list of records based on a human‑readable query like age > 30 and city = "NY". Writing repetitive parsing code in plain Racket can be tedious, but a tiny DSL can make the intent crystal clear.

Step 1: Define the Syntax with Macros

We’ll create a #lang query language that translates the DSL into ordinary Racket predicates. The core macro, define‑query, takes a DSL expression and expands it into a lambda that can be used with filter.

#lang racket

(require (for-syntax racket/base
                     syntax/parse))

;; The macro that turns DSL into a predicate
(define-syntax (define-query stx)
  (syntax-parse stx
    [(_ name:ident expr:expr)
     #'(define name
         (lambda (record)
           (syntax-parse #'expr
             [(?op:id left right)
              (case #'?op
                [(and) (and (eval left record) (eval right record))]
                [(or)  (or  (eval left record) (eval right record))]
                [(>)  (> (field left) (field right))]
                [(=)  (equal? (field left) (field right))]
                [_ (error "Unsupported operator")])])))]))

The macro uses syntax/parse to pattern‑match the DSL operators (and, or, >, =) and generate the corresponding Racket code. Notice how we defer the actual field lookup to a helper function field, which we’ll define next.

Step 2: Helper for Field Access

;; Assume each record is a hash
(define (field expr)
  (cond
    [(symbol? expr) (hash-ref record (string->symbol (symbol->string expr)))]
    [(number? expr) expr]
    [(string? expr) expr]
    [else (error "Invalid field expression")]))

;; Example data
(define people
  (list (hash 'name "Alice" 'age 28 'city "NY")
        (hash 'name "Bob"   'age 35 'city "SF")
        (hash 'name "Carol" 'age 42 'city "NY")))

Now we can declare a query in our DSL and use it directly with filter.

Step 3: Using the DSL

(define-query senior-ny
  (and (> age 30) (= city "NY")))

(filter senior-ny people)
;; → (list (hash 'name "Carol" 'age 42 'city "NY"))

That’s it—one line of DSL replaces a multi‑line predicate function. The readability boost is immediate, especially when the same pattern appears across many modules.

Pro tip: Keep DSL macros pure; avoid side effects during expansion. Pure macros are easier to reason about and compose with other languages.

Case Study: A Mini Language for Financial Calculations

Financial analysts often need to express calculations like “compound interest over N periods” or “net present value” in a concise, spreadsheet‑like syntax. Let’s build a #lang finance language that introduces two new forms: interest and npv. The goal is to let domain experts write formulas without diving into Racket’s numeric library.

Designing the Syntax

• (interest principal rate periods) – Computes compound interest.
• (npv rate cashflows) – Calculates net present value given a discount rate and a list of cash flows.

Both forms will be compiled into efficient Racket functions. We’ll also demonstrate how to extend the language with custom numeric literals for percentages.

Reader Extension for Percent Literals

Racket’s reader can be instructed to treat a token ending with % as a decimal. This makes 5% automatically become 0.05.

#lang racket

(require (for-syntax racket/base
                     syntax/readerr
                     racket/format))

(define-syntax (percent-reader stx)
  (syntax-parse stx
    [(_ token:str)
     (define txt (syntax-e #'token))
     (when (string-suffix? txt "%")
       (define num (string->number (substring txt 0 (- (string-length txt) 1))))
       (unless num
         (raise-syntax-error 'percent-reader "Invalid percent literal" #'token))
       #'(exact->inexact (/ num 100)))]))

;; Register the reader
(begin-for-syntax
  (read-accept-language 'percent-reader))

After loading this module, any file that #lang finance will inherit the percent literal support.

Macro Implementations for interest and npv

#lang racket

(require (for-syntax racket/base
                     syntax/parse))

;; Compound interest macro
(define-syntax (interest stx)
  (syntax-parse stx
    [(_ principal:expr rate:expr periods:expr)
     #'(* principal (expt (+ 1 rate) periods))]))

;; Net present value macro
(define-syntax (npv stx)
  (syntax-parse stx
    [(_ rate:expr (cashflows ...))
     #'(let loop ((cfs (list cashflows ...)) (n 0) (sum 0))
         (if (null? cfs)
             sum
             (loop (rest cfs) (add1 n)
                   (+ sum (/ (first cfs) (expt (+ 1 rate) n))))))]))

Both macros expand at compile time, meaning there’s no runtime overhead for parsing the DSL. The npv macro uses a simple recursive loop to apply the discount factor to each cash flow.

Putting It All Together

#lang finance

;; Using the percent literal
(define principal 10000)
(define rate 5%)          ; becomes 0.05
(define years 10)

;; Compute future value
(define future (interest principal rate years))
;; → 16288.945...

;; Cash flow series for a project
(define cashflows (list -5000 2000 3000 4000 5000))

;; Compute NPV with the same discount rate
(define project-npv (npv rate cashflows))
;; → 1354.23...

With just a few lines of macro code we’ve turned financial formulas into first‑class language constructs. Analysts can now write calculations that read like textbook equations, reducing the cognitive gap between domain knowledge and implementation.

Pro tip: When exposing DSLs to non‑programmers, prioritize clear error messages. Wrap macro expansions in with-handlers to translate syntax errors into domain‑specific hints.

Advanced LOP Techniques: Staging and Compilation

Beyond simple macros, Racket supports staging—generating code that itself generates code. This is useful when you need to compile a DSL into a highly optimized form, such as converting a query DSL into SQL or generating GPU kernels.

Staged Compilation with syntax-local-value

We’ll sketch a tiny example where a DSL for matrix operations is compiled into native Racket loops. The DSL looks like:

(mat-add A B C)   ; C = A + B
(mat-mul A B D)   ; D = A * B

During macro expansion we generate a function that performs the operation using low‑level loops. The generated code is cached using syntax-local-value so repeated uses don’t re‑compile.

#lang racket

(require (for-syntax racket/base
                     syntax/parse
                     racket/syntax))

;; Cache for compiled kernels
(define-for-syntax compiled-kernels (make-hash))

(define-syntax (mat-add stx)
  (syntax-parse stx
    [(_ a:id b:id out:id)
     (define key (list 'mat-add (syntax-e #'a) (syntax-e #'b) (syntax-e #'out)))
     (unless (hash-has-key? compiled-kernels key)
       (hash-set! compiled-kernels key
                  #'(lambda (A B C)
                      (for ([i (in-range (vector-length A))])
                        (vector-set! C i (+ (vector-ref A i) (vector-ref B i)))))))
     #'((hash-ref compiled-kernels key) a b out))]))

When the program runs, the first call compiles the kernel; subsequent calls reuse the cached lambda, achieving near‑C performance for repetitive matrix work.

Integrating with External Tools

Staged code can also emit source files for other languages. For instance, a DSL that describes data transformations can be compiled to Python pandas code, enabling seamless interop in mixed‑language pipelines.

#lang racket

(require (for-syntax racket/base
                     syntax/parse
                     racket/file))

(define-syntax (to-pandas stx)
  (syntax-parse stx
    [(_ (select cols ...) (filter expr) (group-by keys ...) (agg agg-fn))
     (define py-code
       (format "df[~a].loc[~a].groupby(~a).agg(~a)"
               (list->string (list cols ...))
               expr
               (list->string (list keys ...))
               agg-fn))
     (begin
       (displayln py-code)
       #'(void))]))

Running the macro prints the generated Python snippet, which can be copied into a Jupyter notebook. This pattern demonstrates how LOP can act as a bridge between Racket’s expressive macros and the broader data‑science ecosystem.

Pro tip: Use syntax-local-value sparingly; over‑caching can lead to memory bloat. Clear caches when you know the DSL definitions have changed.

Testing and Debugging Your Languages

Debugging macros can feel like peering into a black box, but Racket provides several tools to make the process transparent.

• syntax‑debugger: Visualizes macro expansion step‑by‑step.
• make‑rename‑transformer: Helps avoid identifier capture.
• define‑syntax‑rules vs. define‑syntax: Choose the simpler pattern‑matching system when you don’t need full power.

Example: Inspecting Expansion

(require syntax/debug)

(define-syntax (debug‑query stx)
  (syntax-parse stx
    [_ (begin
         (printf "Expanding: ~a\n" (syntax->datum stx))
         #'(void))]))

Insert (debug‑query …) anywhere in your DSL file to see the raw syntax object printed during compilation. This is invaluable for catching mismatched patterns early.

Unit Testing DSLs

Racket’s rackunit works perfectly for DSLs. Write tests that compare the macro‑expanded code to an expected lambda.

(require rackunit)

(test-case "interest macro expands correctly"
  (define expanded (syntax->datum #'(interest 1000 0.05 3)))
  (check-equal? expanded '(* 1000 (expt (+ 1 0.05) 3))))

By testing the expansion rather than the runtime result, you verify that the DSL’s translation logic stays correct as you evolve the language.

Best Practices for Sustainable Language Design

Designing a language is a marathon, not a sprint. Below are habits that keep your DSL maintainable and enjoyable for collaborators.

1. Start Small: Implement one core feature, get feedback, then iterate.
2. Document Syntax: Use #%module-begin to provide a usage guide that appears when users run raco docs.
3. Separate Concerns: Keep parsing, semantics, and code generation in distinct modules.
4. Version Your Language: Prefix your #lang name with a version number (e.g., #lang mydsl/v1) to avoid breaking downstream code.
5. Provide a REPL: A small interactive shell helps users experiment with the DSL without writing full programs.

Pro tip: Leverage raco make to pre‑compile your language modules. This reduces load time for large DSL projects and catches syntax errors early.

Conclusion

Racket’s blend of a powerful macro system, first‑class language support, and a rich ecosystem makes it uniquely suited for Language‑Oriented Programming. By crafting tiny, purpose‑built DSLs—whether for querying data, performing financial calculations, or generating high‑performance kernels—you can dramatically improve code clarity, reduce boilerplate, and empower domain experts to express their ideas directly in code.

Remember to start with clear, minimal syntax, use staging and caching wisely, and lean on Racket’s debugging tools to keep your language definitions transparent. With these practices, you’ll be able to evolve your DSLs alongside your projects, turning Racket into a living laboratory for language innovation.