Meetups/Infra/2026-03-30

The meetup covered rate limiting strategies for the wiki using Caddy (prompted by scraper-driven DDoS issues), a deep dive into BGP and internet exchange infrastructure, and Jet's ongoing Nix-based wiki rebuild with CI/CD and secret management via SOPS.

• Action Items / Interest Areas

• • Many attendees want to explore getting a Noisebridge box at an internet exchange (Jet, Bjorn, Daniel, Derek, Elan, Ellie, Alex, and others)
  • Wiki hackathon coming up next month
  • Database backups and fulltext storage worth exploring

A closer look at a BGP anomaly in Venezuela https://blog.cloudflare.com/bgp-route-leak-venezuela/

Introductions

• [name] - [background]. [goals for meetup, or interests to explore]

• Loren - he/him - Background cloud engineering, how the wiki works and BGP and networking basics
• Daniel - he or him - fundraising stuff, web development
• Braelyn - like to build cool things, have a corporate job too
• Ellie - likes terminals and databases, wants to hear about BGP
• Dave - works with infrastructure
• Alex - likes distributed systems and botnets, looking for good solutions, if you have 1000 nodes how do you make sure not to duplicate effort
• elan - kubernetes, BGP, has done PGP :D
• Ciara - k8s stan, not
• Chris - web scraping
• Doug - still likes computers, hasn't has had a proper job for >25 years
• Derek - building a website to track my job hunt.
• Jet - they / them, nixxed rusted, oxydized, taking obsession to next level by making them production

• We/Z - all the above pronounce, opencode, loves learening things, ableton production, synths, fpgas, game developer, loves playing with computers.
• Robert - has background in cloud, AI, SaaS, BGP curious,
• Xander - rust-, nixos-, hardware-guy use nixos all thetime for deploying system, curious about networking.
• victor - nix sycophant, all things infra

Lesson or Demo

• Read aloud: clarify for meetup. We are taking notes in a riseup pad (or I am--help appreciated, and links). We have meeting notes posted to the wiki. noisebridge.net, search Infra, or Meetups/Infra. (the Infrastructure page has a disambiguation link.)
• Shell, web services, self-hosting, networking!

wiki topics caddy rate limits

• bgp
  • cloudflare magic

fb outage

• jet — nix wiki
• dave - (demo something)
• alex - k8s — resource sharing 1000 node cluster

- robert - short — readme demo

Review topics planned, Dave got pizza! Thanks Dave!

- Sharing the goods about implementing rate limiting for the wiki, ansible config drift with claude, effective immediately.

- Rate limiting with Caddy

 some paths are going to hit the database hard, cause more load, 
   are more expensive to serve.

- Is there a profiling

 scraper traffic is DDoS'ing wiki.

- caddy with auto request SSL certs - @bot matcher, if "bot" appears in the request header, then the IP is classified as a bot. - google bot is a bit respectful - when there are hundreds of requests per minute, the majority are 500's then don't trust to rate limit themselves. - first zone mataching bot, fastcgi indication for blocks - don't block LAN traffic - rate limit bucket on /24 - session cookie rules, - this on m3 - m3, m4, m6, m7 machine - with this work? defending against traffic - we have a single metric directly from Caddy mysqld phpd and caddy crash scraping full history wrecks the cache load the visual editor, can scrape the raw html. There is logging for these paths, dump to logs Does Caddy support dynamic IP block? Hidden URLs to poison links to nuke bots. Mister Name, google beyond corp, zero trust Zero Trust, authentication on every internal service Insider threats, "Override Security" button creates attack vectors. Logistic regression, for blocks on Caddy files. may serve well as discussion. - Glances, python tool.

https://github.com/noisebridge/infrastructure/issues/472

• bgp

 MagicDNS from proxy traffic through cloudflare, cloudflare controls nameservers though edge network via anycast
 what is anycat, not multicast.
 a service with multiple machine with load balances, one of many servers take ownership.
 Cloudflare can be configured for DDoS absoroption, if your IP doesn't have a strict allowlist, the if someone finds your IP, they can still DDoS you. 
 ICANN
 - root of the heirarchy
 - ipv4, ipv6, CIDR, domain names
 - AS, autonomous systems.
   distributed
   choose to peer
   gossip, learn about routes, which networks have which IPs
   collescing time, route to all IP addresses
   How is trust established?
   Issues with, there are now internet exchanges, which AS's are allowed to 
   Pakistan, et al, started announcing bunk IPs, knocking services offline
 - Cloudflare AS
 - Tier 1 providers
   filter lists 
   RIPE, internet protocol european
   internet exchange, SFMix, local network bits
   (SFMIX has a nice graphana)
https://grafana.sfmix.org/public-dashboards/e93a968eb538461da4c6ada750b33495?orgId=1&refresh=10s

https://bgp.tools/ixp/FCIX
https://www.peeringdb.com/ix/2163
https://search.brave.com/search?q=ipv4+routing+table+size&source=web&summary=1&conversation=08e9f4cb7d3d3c0f5323a2f199de0327f9eb

https://www.cidr-report.org/as2.0/

https://fcix.net/services/
https://sfmix.net/

   Quick Question:
       entry point to finding a computer
       domains are given by registrars
       DNS domain -> IP
       registrars get to charge for administration access to network
       when you want to an IP, get one from ARIN in US, RIPE in Europe
       registrars are weird gate keepers on the highway
       BGP gossips minimal routing table
       ISP has leased addresses, 
       Different people aggregate different routes together, contiguous range of ips
       Fremont network Cabal
         Have a low ASN, because an architecture got their own website in the 80's
         Dentists were on it!
       Registering with ARIN is $100, just part of the internet craze to make
       Announced Router Vendor
       KPFA peering exchange point
   Sonic: Could we get fiber at noisebridge?
     Small Business setup, email the founder, get a network engineer here, 
   
   What was the anatomy of the facebook BGP world outage?
     facebook hosts a lot of internet
     somewere in the multilayers mess, work hard to not make it a mess
     there were configuration sensitive
     When Iran / Pakistan messes with BGP, and because they the authoritative source
       the system hardlocked
       resiliancy?
       may have migrated domains
   Internet Exchanges, what is it?
     They often have route servers, do the calculatiion
     do the routing summaries,
     choose to peer with networks, some routes are announced publically
      there are business agreement
       high capacity switch 10GB Ports
       Trade bandwidth, have different
       netflix had high percent of network bandwidth
         how to build seed boxes, pop nodes, point-of-presence
 Anycast
   have an AS, with IP, Quad 1, want to serve to India. seek shortest gossip route
   have traffic routes here, and half routes there,
   shortest logically, route flapping   
     ameliorate, cloudflare can serve different locations 
       by different servers based on locality,
     mobile phone networks keep traffic in network, 
       regional exits, 
     least cost routing?
       going off network, then they do pay
 
 Robert : $10K Bounty?
   speed running v8 js mcp server
   agents, in order to be useful, they need access to computation
     a full computer is too much, what about isolation?
       only call urls with certain prefix, more finegrained than MiTM proxy
       supports TypeScript
       load WASM
       import npm packages
       common attack vector, allow pypy as package repo, package filter
       example policy
         filter URLs
       Q: read access for everything local, built a good sandbox? FS Read-only
       A: craft policies for read-only access to filesystems, compose MCP tools together, regular v8 for system calls
       lots of agents use blocking for tool calls
       if write a cmdline version for the LLM
       MCP is more accessible to the LLM 
       MCP vs. CLI is modern VIM vs. emacs
       
       Don't use any system calls, uses v8 as runtime
         Call C-API, unveil on directory?
         v8 has runtime javascript
         bug hunt welcome!
         

• Dave:

 Static Rendering for wiki?
 Interesting general problem, blob storage to replace database
 slow cheap version of cassandra, run kafka queues
   For systems that are easy to copy?
   The approach of static rendering is a plugin
   Will post later
   Wiki hackathon is happening in the next month.
     
 

• Jet

 Noisebridge, large monorepo (400K lines).
   from the beginning
   Lots of config files
   deploys to m3...
   deploys a bunch of different services
     interesting repo to inspect
     wiki was getting difficult to maintain in some aspects
     still operating on a VM with 1 core, 2GB, no replicas
     config through ansible, with php configs, 
     CI/CD for the entire repo
       to test entire config? doesn't make a lot of sense
       To the wiki
 Wiki, let's trying rebuilding the wiki,
   The goal? Make the noisebridge infra
   Accidently used mediawiki 1.45, then back to 1.39 without changing integration
   Ansible starts with downloading 1.39
     If that install happens with docker snapshot
     ( Do databases support git like versioning)
     The infrastructure is deterministic nix.
     github workflows?
       pr-check: open a ubuntu, nix run check, nix flake check across entire build
         based on different architecture,
           two machines.
             - main production box.
             - replica wiki, read-only, sync'd only a few seconds behind 
       main-deploy: 
         check everything, building everything, deploy-rs
           imageine k8s combined with nix at the same (crazy nerdy shit)
           like docker swarm,
       multiple admins?
       all of the succesful deployments
       weird migration mistake
       features that we can have
         Secret Management?
         Separate vault for keys, switched from file based configuration
         share single master keys
           switch to using SOPs, agai-nix commit secrets directly to repo
           when machine starts up, it will take own private key
           decrypt secrets, 
             public keys of admins,
             secrets are sync'd to commits
   The goal of the project is to make the wiki itself singular thing
   separating out the heavy thing, it occupies a large space.
   State drift is a major concern
   All the image assets copied to T, 
   Go onto new server, with replication server, in SF.
   Adding extensions good exercise
   m3 pays $15 / mo to hurricamne electric 2 cents. 
   
   Is the wiki containerized?
     nix the configuration, and state, marina db, assets images and files.
     
   Alex, 1000 ec2 instances, inside agent container
   by degault need to load updates on 
   how to manage the pipeline
   Facebook developer blog, mounts encrypted, deploys software as ISO, mounted on loopback with clear story about provenance
   

 - bgp.tools
 - AS number. 
 ARPA
 ARIN
 

• • cloudflare magic
  • fb outage

Outros

• We-z - sfmix!
• Robert - BGP
• Victor - needs to brush up on networkd
• Robbie - BGP, brush up on
• erik - stalled deno, doesn't know enough about it.
• Xander - listen to oxide and firneds podcast who setup freemont cabal
• Robert - nothing interesting tonight, nix is interesting
• Braelyn - Appreciate the lower level
• Eliie - running an internet exchance is more accessible than thought
• Bjorn - like to learn more about nix
• TJ - how would you contribute to noisebridge how to test locally
• Elan - not that into nix, but supports it. backup story? provisioning nix things in the cloud, digital ocean
• Dave - missed the BGP talk, jet's work is greate
• Chris - pass
• Doug - really enjoyed looking at a well formed yaml file, looking forward
• Derek - like the network material, really like that stuff, network eng stuff
• Ciara - dive into jet's project
• Jet - really badly wants to get a box in an internet exchange Derek BJorn,
• Alex - also in on the box, and k8s big talk

Who wants to explore getting a box in an internet exchange: Jet Bjorn Daniel Derek Elan Ellie Zacchae Alex We/Z

On the metal, PhirePhly

If the noisebridge noisegarden can we run quizbowl software for infrastructure stuff Trivia software is seriously good, written by the pub owner.

would be cool if we had database backups if we stored fulltext

https://www.youtube.com/watch?v=qJo_b3Euxes

Questions, Discussion, or Coworking

• [Issue]

For next time

Readings & Exercises

• Readings
• Exercises

Join online

• Try it yourself!
  • Join libera.chat #nb-meetup-infra

https://www.noisebridge.net/wiki/Meetups/Infra