Skip to content

backend.md

Latest commit

 

History

History
317 lines (219 loc) · 23.9 KB

backend.md

File metadata and controls

317 lines (219 loc) · 23.9 KB

Backend Architecture

Stack: Django 6.0 + Django REST Framework 3.17 + PostgreSQL 18, served via Daphne (ASGI).

The backend is a thin API layer for CivicTechJobs's owned domain: a skill catalog, the volunteer qualifier flow (where users record skills with proficiency levels and supply availability), opportunity listings, and the matching algorithm that connects volunteers to opportunities.

Django admin doubles as a project-manager-facing CMS for opportunity management and a management interface for the skill catalog; there is no separate admin UI.

Stage taxonomy

The rewrite is split into stages. This doc describes Stage 1 in detail and stubs Stage 2 for forward-compatibility.

  • Stage 1 (current PR scope) - Django default User-based auth, locally-curated skill catalog, no Cognito, no PeopleDepot integration. Goal: a working MVP without external dependencies.
  • Stage 2 (deferred) - Cognito JWT authentication (ID-token), with reference data (user identity, practice areas, roles, projects) sourced from PeopleDepot at request time and the skill catalog synced from PeopleDepot into a local cache. Stage 2 is gated on PeopleDepot's prod-deployment posture, which is in flux upstream (PeopleDepot Issue #218).

In Stage 2 the user-facing model is unchanged: a volunteer still selects skills with proficiency levels. What changes is that the catalog of available skills is synced from PeopleDepot rather than curated locally. CTJ keeps a local Skill cache (for query performance and resilience) and continues to own the user's selections (the SkillMatrix).

Project layout

backend/
├── backend/                   # Django project config (settings, root routing, ASGI)
│   ├── settings.py
│   ├── urls.py
│   ├── views.py               # Healthcheck + JSON 404 for unknown /api/* routes
│   ├── asgi.py
│   ├── wsgi.py
│   └── templates/
├── ctj_api/                   # Main Django app - CTJ-owned domain
│   ├── models.py
│   ├── views.py
│   ├── serializers.py
│   ├── permissions.py
│   ├── urls.py
│   ├── admin.py               # PM/admin-facing CMS configuration
│   ├── auth.py                # Cognito JWT auth backend (Stage 2 stub)
│   ├── clients/
│   │   └── peopledepot.py     # PeopleDepot API client (Stage 2 stub)
│   ├── migrations/
│   └── tests/
├── manage.py
├── pyproject.toml             # Poetry-managed dependencies + ruff/mypy/bandit config
├── poetry.lock
├── openapi-schema.yml         # Auto-generated by DRF
├── entrypoint.sh              # Docker entrypoint for stage / prod
└── startServer.sh             # Run the server outside Docker

backend/backend/ is the Django project (config + routing). backend/ctj_api/ is the only Django app - all CTJ-owned domain logic lives there.

auth.py and clients/peopledepot.py are placeholders for Stage 2; they will be added when Cognito + PeopleDepot integration lands and are not part of Stage 1.

Data model

CTJ's domain models. The Skill table is locally curated in Stage 1; in Stage 2 the catalog source moves to PeopleDepot while CTJ continues to store the per-user SkillMatrix selections.

  • Skill - skill catalog. Fields: name (unique), description, and an array of practice-area references. In Stage 1 the catalog is curated through Django admin. In Stage 2 it becomes a local cache synced from PeopleDepot - the table stays, but it's populated from PD rather than hand-curated, and admin gains read-only views instead of CRUD.
  • SkillMatrix - JSON field mapping {skill_uuid: mastery_level (1-5)}. Skill UUIDs reference the catalog (locally-curated rows in Stage 1, PD-sourced rows in Stage 2). Shared between users (skills they have) and opportunities (skills they require) - it's the data structure the matching algorithm operates on. JSON over a join table because ratings are sparse and the matching algorithm operates on the whole structure as a unit.
  • Opportunity - open volunteer position. Fields: body (description), required skills (O2O to SkillMatrix), status enum (open / closed / on hold / filled / draft), created_by (FK to UserProfile), and CTJ-specific fields (work environment, minimum experience, minimum hours). Stage 1 carries local project and role FKs; Stage 2 swaps both to PeopleDepot UUID references.
  • UserProfile - AUTH_USER_MODEL, an AbstractUser subclass. Fields beyond the AbstractUser defaults: community-of-practice selection, O2O to a SkillMatrix (skills_learned_matrix), availability (max_available_hours, meeting_availability), is_project_manager flag. AbstractUser over a O2O profile extension keeps user fields on one model rather than dual-table joins, and the Stage 1→2 migration is cleaner. In Stage 2, UserProfile gains a cognito_sub field (also the PeopleDepot user UUID) and identity fields (name, email) are sourced from PeopleDepot rather than locally edited.

API endpoints

Endpoint Methods Auth Notes
/api/healthcheck/ GET None Uptime + version
/api/skills/ GET None Skill catalog (Stage 1: locally curated; Stage 2: locally cached from PeopleDepot)
/api/communities-of-practice/ GET None CoP taxonomy (locked to 5 values)
/api/roles/ GET None Role catalog
/api/projects/ GET None Project list
/api/opportunities/ GET, POST GET: None, POST: PMs List / create
/api/opportunities/{id}/ GET, PUT, DELETE GET: None, PUT: creator, DELETE: any PM Detail / update / delete
/api/users/<uuid:pk>/ GET Authenticated, self only Per-user record

The matching endpoint (ranking opportunities against a user's SkillMatrix or vice versa) is part of Stage 1 work - see What's not built yet.

The OpenAPI spec is at backend/openapi-schema.yml and is regenerated whenever startServer.sh runs.

Non-existent /api/* routes return a structured JSON 404.

View shape

DRF gives several ways to write a view (ModelViewSet, ReadOnlyModelViewSet, single-purpose generics like RetrieveAPIView, function-based views with @api_view). Each hides different amounts of behavior. To keep views.py readable cold, this project uses two view shapes:

Endpoint shape View shape
Full CRUD on a resource (>=4 of list/create/retrieve/update/destroy) ModelViewSet registered on DefaultRouter in backend/ctj_api/urls.py
Anything narrower (single method, list-only, retrieve-only, list+retrieve, custom action) Function-based view decorated with @api_view([...]) and @permission_classes([...]), routed explicitly with path()
Non-resource (health, fallbacks) Plain Django def view(request)

The rule is intentionally simple: any view that doesn't earn the full CRUD surface stays an FBV so the method, URL, and permission policy are visible inline at the function. OpportunityViewSet is the only ModelViewSet in the codebase; everything else is an FBV or a plain Django view.

When converting a viewset to FBVs, list and detail become two separate functions (e.g. skill_list and skill_detail). Each gets its own path() entry in backend/ctj_api/urls.py. The router only carries the ModelViewSets.

One detail worth flagging: object-level permission classes (has_object_permission) are auto-triggered by generic-view internals, but not by @api_view. An FBV that uses a permission class with object-level checks needs to call request.parser_context["view"].check_object_permissions(request, obj) explicitly after the object lookup. See user_detail in backend/ctj_api/views.py for the canonical example.

URL routing

Resource URLs are kebab-case, plural, with a trailing slash. Non-resource endpoints (health, fallbacks) follow the same casing/slash rules but stay singular because they don't represent collections.

Aspect Rule Example
Word casing kebab-case (- between words) communities-of-practice, forgot-password
Pluralization Plural for collection endpoints; singular for non-resource /api/skills/, /api/healthcheck/
Trailing slash Always present /api/skills/, not /api/skills
Path parameters UUID via <uuid:pk>; named when multi-segment (<uuid:project_id>) /api/users/<uuid:pk>/

Underscores in URLs are avoided (kebab over snake) — URLs are protocol-level path segments, not Python identifiers; the HTTP/REST tradition is kebab-case, and underscores can disappear under hyperlink underline. Trailing slashes are always present so Django's APPEND_SLASH=True default doesn't issue a 301 redirect on every request.

If a route legitimately needs to deviate (a webhook endpoint where a third party requires a specific shape), flag the deviation in the URL config docstring.

Serializer shape

Each resource gets a separate XxxReadSerializer and XxxWriteSerializer when both shapes are needed. Read-only resources only have a Read serializer; a Write serializer is added when (and only when) a write endpoint is added. This keeps the input/output contract entirely at the serializer layer rather than splitting it between the serializer (fields) and the view (allowed methods).

Resource state Serializer classes
Read-only (no write endpoint exists) XxxReadSerializer only
Read + write XxxReadSerializer and XxxWriteSerializer

Auto-managed fields (id, created_at, updated_at) and request-stamped fields (e.g. Opportunity.created_by, set from request.user by the view) are absent from XxxWriteSerializer.Meta.fields rather than included with read_only=True. The absence is the contract: if the field doesn't appear in Meta.fields, the client cannot supply it.

Read/write divergence (different exposed fields, computed-on-read fields like OpportunityReadSerializer.created_by stringified to email, validation only on write) becomes a class boundary rather than per-field flag manipulation. Future changes to write behavior land in XxxWriteSerializer only; read responses are untouched.

OpportunityViewSet (the one ModelViewSet in the codebase) dispatches to the right serializer via get_serializer_class():

def get_serializer_class(self):
    if self.action in ("list", "retrieve"):
        return OpportunityReadSerializer
    return OpportunityWriteSerializer

FBVs reference the right serializer directly — they only do one thing, so they only ever need one class.

Error envelope

All error responses share a single shape:

{
  "error": {
    "code": "<machine_readable_snake_case>",
    "message": "<human-readable string>",
    "fields": { "<field>": ["<msg>"] }
  }
}
  • error is always an object, never a string.
  • error.code is a snake_case identifier; clients switch on it. Stable across translations.
  • error.message is a human-readable string suitable for displaying to users when no field-level surface is appropriate.
  • error.fields is present only for validation errors. Keys are field names; values are arrays of messages. Top-level (non-field) validation errors land under the non_field_errors key, mirroring DRF's existing convention.
  • HTTP status code lives only in the response status; it is not duplicated in the envelope body.

DRF-raised exceptions are wrapped through ctj_api.exceptions.civic_exception_handler (registered as REST_FRAMEWORK["EXCEPTION_HANDLER"] in backend/settings.py). The handler maps each DRF exception class to a code:

Exception error.code
ValidationError validation_error (with error.fields)
AuthenticationFailed authentication_failed
NotAuthenticated not_authenticated
PermissionDenied permission_denied
NotFound not_found
MethodNotAllowed method_not_allowed
NotAcceptable not_acceptable
UnsupportedMediaType unsupported_media_type
ParseError parse_error
Throttled throttled
(other DRF APIException) error

Manual error paths (the api_not_found catch-all for unknown /api/* routes is the current example) construct the envelope inline because they don't go through DRF's exception machinery — they're Django URL fallbacks. Use the same shape.

When a new error path is added, prefer raising a DRF exception (ValidationError, PermissionDenied, etc.) so the handler renders the envelope automatically. Construct the envelope inline only when raising would be the wrong tool (e.g. URL-level catch-alls, asynchronous task error responses).

Test shape

Tests live in backend/ctj_api/tests/, one file per resource:

ctj_api/tests/
├── common.py                       — factory helpers, no test classes
├── test_healthcheck.py
├── test_users.py
├── test_opportunities.py
├── test_community_of_practice.py
├── test_roles.py
├── test_skills.py
└── test_projects.py

Each file holds one <Resource>Tests class extending APITestCase. Each class has its own setUp that constructs only the rows its tests need — there is no shared APITestBase because there is no meaningful universal setup (healthcheck needs nothing; user-detail needs users; opportunity tests need users + reference rows).

Shared fixture-construction lives in common.py as factory functions (make_pm_user, make_regular_user, make_cop, make_role, make_skill, make_project, make_opportunity). Each call returns a saved instance; defaults are sensible and overridable via keyword arguments. The factories don't share state — each call creates a new row.

Test method names follow test_<subject>_<action>_<expectation>:

  • subject — who's making the request (anonymous, regular_user, pm_user, authenticated_user)
  • action — what's being attempted (list, create, view_own_record)
  • expectation — the result (returns_200, cannot_create, gets_403)

Examples: test_anonymous_can_list_opportunities, test_regular_user_cannot_create_opportunity, test_authenticated_user_cannot_view_others_record. Each test method also has a one-liner docstring describing the assertion in plain English.

When a resource grows write tests alongside read tests (and the file gets long), split into multiple classes within the same file: <Resource>ReadTests, <Resource>WriteTests. Don't split into separate files unless the test count justifies it.

Auth

Stage 1 - Django's default session authentication. The DRF API uses SessionAuthentication; the app frontend signs in through a Django-issued session cookie. createsuperuser is the bootstrap path for admin / PM accounts. Sessions over token auth here because Django admin already uses sessions, so reusing them keeps Stage 1 free of extra auth infrastructure.

Stage 2 (deferred) - Cognito JWT (ID-token), validated by a custom DRF authentication backend at backend/ctj_api/auth.py (added when Stage 2 work begins). The Next.js frontend signs the user into Cognito, holds the tokens server-side, and forwards the ID token in Authorization: Bearer <token> for protected mutations. The backend verifies the JWT signature against Cognito's JWKS, validates iss / aud / exp, and resolves sub to a local UserProfile row. ID-token over access-token because the ID-token carries identity claims directly; the API doesn't need to call Cognito's userinfo endpoint per request.

How a Cognito-authenticated user reaches Django admin (/admin/) in Stage 2 is an open design question, deferred until Stage 2 work begins. Stage 1 doesn't hit this problem because Django admin's native login flow is the auth path.

Permissions

Custom DRF permission classes in backend/ctj_api/permissions.py:

  • OpportunityPermission - public read (no auth required, mirroring /api/skills/); only PMs can create; only the creator can update; any PM can delete. Public read because opportunities are a recruitment catalog - friction-to-browse should be zero, and signup belongs at the apply / register-skills step, not at discovery.
  • UserDetailPermission - /api/users/<uuid:pk>/ is self-only: a user can fetch their own record but not anyone else's. The class only overrides has_object_permission; request-level auth is enforced by stacking IsAuthenticated separately on the consuming view.

Skill catalog mutation is gated by Django admin's built-in staff permission, not a separate DRF class - the API only exposes GET /api/skills/.

PM status comes from the isProjectManager flag on CustomUser. Existing admins set the flag through Django admin.

Permission class shape

Permission classes override only the layers they need:

  • Override has_permission if any rule depends only on the request (HTTP method, user properties readable without a database lookup, headers).
  • Override has_object_permission if any rule depends on the row being checked (creator-only, organization-membership, status-based).
  • Override both if rules span both layers (this is OpportunityPermission's shape).
  • Override neither and use a built-in (IsAuthenticated, IsAuthenticatedOrReadOnly, AllowAny) when the policy reduces to one of those — custom permission classes are reserved for resource-specific logic.

Class naming: <Resource>Permission for resource-scoped policy. Is<Capability>Permission for cross-resource capabilities (none in the codebase yet, but reserved for the auth rebuild).

Method-body conventions:

  • Early-return branches rather than nested ifs. Each method ends with an explicit return False for fall-through.
  • Use getattr(request.user, "attr", default) for fields that anonymous users don't have — direct attribute access raises AttributeError on AnonymousUser.
  • has_object_permission should exempt SAFE methods (if request.method in permissions.SAFE_METHODS: return True) only when the resource is publicly readable. For self/owner-only resources where reads are also gated, do not exempt SAFE methods — the identity check handles all methods.

permission_classes composition (in views):

  • Tuple for class-level attributes: permission_classes = (IsAuthenticatedOrReadOnly, OpportunityPermission).
  • List for the @permission_classes([...]) decorator on FBVs.
  • Order from least-to-most-specific: built-in DRF gates first (auth), resource-specific custom classes after. Reading top-to-bottom reads the gate progression.

Django admin as CMS

Django admin at /admin/ is the management interface for two CTJ-owned domains:

  • Opportunities - PMs create and update opportunities directly.
  • Skill catalog - admins curate the list of skills (add, edit, remove), set descriptions, and associate skills with practice areas. (Stage 1 only - in Stage 2, the catalog is synced from PeopleDepot and CTJ admin's role narrows to read-only inspection.)

Django admin (rather than a custom CMS) gives free CRUD scaffolding for the PM-facing interface; building a custom UI would be cost without product gain.

In Stage 1, admins log in through Django admin's standard username/password flow. The first admin is created with python manage.py createsuperuser. Subsequent admin and PM elevations happen through Django admin's UI.

Reference data outside CTJ's scope - practice areas, roles, projects, user identity - is not manageable through CTJ's admin. Those become PeopleDepot-owned in Stage 2.

Frontend integration

In Stage 1, the Next.js frontend talks only to this Django API. Server components and server actions in Next.js read CTJ data (skill catalog, opportunities, skill matrices, matching results, user profile) and submit qualifier-flow updates through the same API. The browser never talks to the Django backend directly - Next.js owns the request boundary.

In Stage 2, Next.js will additionally talk to PeopleDepot for reference data (practice areas, roles, projects, user identity). The skill catalog stays behind CTJ's /api/skills/ endpoint - CTJ caches it locally and syncs from PeopleDepot. Server components aggregate both sources in a single render pass.

There is no frontend_dist/ build copy. The Next.js container is deployed alongside Django in the same ECS task; see deployment-infra.md for the deployment topology.

PeopleDepot integration (Stage 2)

PeopleDepot's prod posture is in flux upstream, so Stage 2 is held until the integration surface settles. The intended shape:

  • User identity - Cognito subjects map 1:1 to PeopleDepot user records. Name, email, and basic profile come from PeopleDepot.
  • Skill catalog - PeopleDepot publishes the master list. CTJ keeps a local cache of the catalog (synced from PD) so the qualifier flow stays fast and resilient. Users select from the cached catalog; CTJ stores their selections (the SkillMatrix is keyed by PeopleDepot skill UUIDs).
  • Practice areas (Communities of Practice) - taxonomy and descriptions sourced from PeopleDepot.
  • Role / job-title taxonomy - PeopleDepot's term is ModernJobTitle.
  • Project metadata - project name, meeting times, status, etc.

Implementation is a PeopleDepotClient interface at backend/ctj_api/clients/peopledepot.py (added when Stage 2 work begins). Methods will be shaped after PeopleDepot's published OpenAPI schema (generated by drf-spectacular).

The Stage 1 → Stage 2 migration drops local reference-data tables (Project, Role-equivalent) and adds cognito_sub to UserProfile for matching against PeopleDepot user records. The Skill table is retained as a synced cache rather than dropped.

What's not built yet

Stage 1:

  • Matching algorithm - backend logic and endpoint shape for ranking opportunities by SkillMatrix similarity.
  • Opportunity filtering / search - list endpoint currently returns all opportunities with no query params.
  • PUT /api/users/me/ - needed so the qualifier flow can write the user's CoP selection, skill matrix, and availability.

Stage 2:

  • Cognito JWT authentication backend in auth.py.
  • PeopleDepotClient implementation against the live API.
  • Skill catalog sync from PeopleDepot (populates the local Skill cache).
  • Data-model migration: drop local CommunityOfPractice / Role-equivalent / Project; add cognito_sub to UserProfile; replace local FKs with PeopleDepot UUID references on Opportunity.
  • Django admin auth bridge for Cognito users (open design question).

Local dev - lint

The backend lint stack is ruff (lint + format + import sort, replacing the legacy black + flake8 + isort chain), mypy for type checking, and bandit for security scanning. Run from backend/:

poetry run ruff check .          # Lint (pyflakes + pycodestyle + isort + bugbear + django + ...)
poetry run ruff format .         # Format (black-compatible)
poetry run mypy ctj_api backend  # Type-check (gradual mode; see CONTRIBUTING.md)
poetry run bandit -r ctj_api backend -c pyproject.toml  # Security scan

All four tool configs live in backend/pyproject.toml under [tool.ruff], [tool.mypy], [tool.django-stubs], and [tool.bandit]. Pre-commit hooks run ruff automatically; CI runs the full suite per PR. See devops.md → Linting.

External docs