xash3d-fwgs/ref/vk/TODO.md

Next

• bounce diffuse is still way darker than before

Previously

2024-01-18 E365

• [-] flashlight far circular glitches
  • This is due to f32 precision not being enough when working with small (light radius ~=1) and large (light distance ~=1e4) numbers.
• patchable sun angle
  • ❌ does qrad already have something for that? → no it doesn't
• cleanup this TODO

2024-01-16 E364

• P NaNs
  • need to remove degenerate triangles
• light_environment is too dark
• add direct_{diff,spec} to rendertests → only can do for this handmade-brdfs branch
  • ❌ and rerun tests for vulkan to get new gold images → imuposshiburu, see above

2024-01-15 E363

• filter out invalid (r=0, etc) lights in native
  • [-] ⭕ already do; it seems that clusters are not getting updates → see #730
• pass point lights r² directly?
• move empirical scaling to native code
• modify point light radius in entity patches → already done
  • adjust brightness based on radius? → already done
• ❌ common intersection-local-normal-oriented basis → point light construct light-oriented frames, not reusable

2024-01-12 E362

• point→spherical light sampling
  • 1/pdf → pdf *= 2π
  • disk sampling

2024-01-11 E361

• fix zero-area polygon lights nanites, fixes #461
  • c1a1a NaNs are still there
• fix point light computation instabilites
  • need proper sampling asap, as different instabilities approaches are visually different, and it's impossible to reason which one is preferable
• add material debug display mode
• vulkan validation layers crashes on too many debugPrintfEXT messages

2024-01-09 E360

• validate all intermediate and final outputs against invalid values, complain into log
• brdf math surprising edge cases
  • alpha^2 == 0 ???
  • various N,L,V collinearities, zero denoms and infinities
    • h_dot_l | h_dot_v < .0 because of numerical precision
    • ggxV|ggxG denoms->0

2024-01-08 E359

• [-] find and fix MORE NaNs
  • add debugPrintfEXT to shaders
  • fix black dots on glass surfaces
  • fix polygon light nans in logs
  • magenta gliches -- dot(N,L) < 0.
  • disableable NaN debugging with macro
  • enable NaN debugging with -vkvalidate

2024-01-04 E357

• Black metals: https://github.com/w23/xash3d-fwgs/issues/666
  • fix missing dot(N,L) term
• try bespoke diffuse term -- yes, mine seems to be more correct
  • PR against glTF
• Bounces
  • idiotic sampling
  • sampling functions
    • diffuse
    • specular
  • how to mix properly with brdf itself
  • find and fix NaNs
• Better PBR math, e.g.:
  • Fresnel issues (esp. with skybox)
  • Just make sure that all the BRDF math is correct

2023-12-29 E354

• Figure out why additive transparency differs visibly from raster
• Implement special legacy-blending in sRGB-γ colorspace

2023-12-28 E353

• track color spaces when passing colors into shaders
• [-] validation failure at startup, #723 -- seems like memory corruption

Longer-term agenda for current season:

• Transparency/translucency:
  • Proper material mode for translucency, with reflections, refraction (index), fresnel, etc.
  • Extract and specialize effects, e.g.
    • Rays -> volumetrics
    • Glow -> bloom
    • Smoke -> volumetrics
    • Sprites/portals -> emissive volumetrics
    • Holo models -> emissive additive
    • Some additive -> translucent
    • what else
• Render-graph-ish approach to resources.
• Performance tools -- needed for perf and lighting work below:
  • Needs: render-graph-ish things for fast iterations when exporting custom situational metrics for the shader.
  • Purpose: shader profiling. Measure impact of changes. Regressions.
  • WIP shader clocks: https://github.com/w23/xash3d-fwgs/pull/692
  • WIP perf query: https://github.com/w23/xash3d-fwgs/pull/500
• Lighting
  • Point spheres sampling
  • Increase limits
  • s/poly/triangle/ -- simpler sampling, universal
  • Better and dynamically sized clusters
  • Cache rays -- do not cast shadow rays for everything, do a separate ray-only pass for visibility caching
• Bounces
  • Moar bounces
  • MIS
  • Cache directions for strong indirect light

2023-12-19 E350

• fixup skybox reflections
• improve logs "vk/tex: Loaded skybox pbr/env/%.*s"
• add skybox test

2023-12-18 E349

• KTX2 cubemaps
• variable cubemap exposure (in .mat file)

2023-12-15 E348

• fix ktx2 sides corruption

2023-12-14 E346-E347

• Optimize skybox loading, #706
  • Do not load skybox when there are no SURF_DRAWSKY, #579
  • Do not reload the same skybox
  • [-] Load skyboxes from KTX2 sides → doesn't work as easily, as there's no way to rotate compressed images. KTX2 sides should be pre-rotated
  • do not generate mips for skybox
  • support imagelib cubemaps
  • use imagelib skybox loader
• Hide all SURF_DRAWSKY while retaining skybox, #579
• possible issues with TF_NOMIPMAP
  • used incorrectly when loading blue noise textures
  • what about regular usage?

2023-12-11 E345

• fix black dielectrics, #666
  • fix incorrect basecolor brdf multiplication, #666
  • fixup skybox glitches caused by #666 fix
• Patch overlay textures (#696) → turned out to be much more difficult than expected.
• Do not patch sprite textures for traditional raster, #695

2023-12-05 E342

• tone down the specular indirect blur
• [-] try func_wall static light opt, #687 → decided to postpone, a lot more logic changes are needed
• increase rendertest wait by 1 -- increased scroll speed instead
• update rendertest images
• Discuss shader profiling
• [-] Discuss Env-based verbose log control

2023-12-04 E341

• [-] investigate envlight missing #680
  • couldn't reproduce more than once
• add more logs for the above
• double switchable lights, #679

-- season cut --

2023-12-01 E340

• Better resolution changes:
  • Dynamic max resolution (start with current one, then grow by some growth factor)

2023-11-30 E339

• rendermode patch
  • track patch by boolean, not another field
• missing polylight on c2a1b
  • "proper" slow fix: make func_water emissive surfaces dynamic
    • extract dynamic polylights from render/rt models
• Reuse GPU scope names
• Support changing screen resolution up to UHD
  • Increase devmem count.

2023-11-28 E338

• rendertest
  • read imagecompare results
  • html report

2023-11-27 E337

• make rendetest.py the central script
  • parallelize/make gifs
  • diff/convert in parallel
• [-] backside transparency
  • added to rendertest
  • consider passing a special flag for single-sided blended surfaces (i.e. brush surfaces)
• fix per-entity material mapping, #669
  • add to rendertest

2023-11-24 E336

• reproducible rendering:
  • make sure it's reproducible -- given carefully spaced wait Ns and playersonly it gets pretty reproducible
  • difference heatmap
  • contemplate infrastructure: scripts, repo, etc.

2023-11-23 E335

• spec for profiler dumper
• reproducible rendering:
  • write fixed resolution internal images -- only need this because i'm stupid and using tiling window manager
    • how to synchronize with frames
    • how to extract vk images
    • how to blit/copy various image pixel formats
    • what file format to choose for non-rgba8 formats? do we even need them?
  • script for running and comparing results
  • [-] extras:
    • difference gif
    • difference summary table
    • summary html
• consolidate all binding in shaders

2023-11-21 E334

• reproducible rendering
  • dump all components
    • script
    • [-] try also dumping in native code -- no need, it's fast enough
  • command for random seed fixation

2023-11-20 E333

• contemplate testing rendered images
  • try making a rendertest script: load multiple save, make multiple screenshots
  • compare screenshots
  • Other infrastructure:
    • tracking golden states
    • testing script

2023-11-17 E332

• [-] backside emissive water polygons:
  • adding them makes things worse in other parts of the level
• water normalmap support -- added missing tangents
• discuss integration test strategies

2023-11-16 E331

• Emissive waters
  • add emissive water surface to polygon lights
  • update emissive color for water surfaces
• trihash option
• dynamic UVs
  • update UVs for conveyors
  • pls don't aggravate validation on changelevel -- cannot reproduce

2023-11-14 E330

• culling worldmodel waters
  • [-] try simple flag culling (probably won't work)
  • [-] try detecting glpoly normals -> consistent with SURF_PLANEBACK, doesn't help
  • SURF_UNDERWATER seems to get us a SINLE surface looking outwards
• investigate gl backface culling for transparent surfaces:
  • glass -- seems to have 2nd face (brush backside)
  • water -- doesn't seem to have 2nd face
    • glpoly_t winding order is reversed when camera origin is opposite to (SURF_PLANEBACK-aware) surface normal
• discuss culling transparent surfaces strategies

2023-11-13 E329

• [-] culling -> need to cull everything except opaque and blend. Alpha-mask is culled.
• [-] waters:
  • [-] No water surface visible from underneath -- hidden by enabling culling
  • [-] No coplanar issues visible? -- hidden by culling. Disabling culling makes glitches reappear

2023-11-10 E328

• woditschka
  • [-] potentially collinear planes vs ray tracing #264
    • not super clear how exactly it works, and what it does. And how to cull things
    • leaning towards making our own tesselator, as it might be universally usable for other things, e.g. detail mapping
    • (A) try producing simple surfaces w/o tesselation, similar to regular brush surfaces
    • (C) print out all surfaces and polys to see where are they looking
    • [-] (B) try filtering surfaces looking down

2023-11-09 E327

• update animated textures is now super slow: some static map surfaces have alternate anims (e.g. light post on c2a5)
• [-] woditschka
  • height not switching to negative underwater -- decided that we don't need it for now
  • do not draw water sides when not requested.

2023-11-07 E326

• list supported arguments for rt_debug_display_only cvar
• make vk_debug_log a command
• remove stvecs from patches -- not used, inconvenient
• patch texture coordinates by matrices
• add _xvk_tex_rotate
• ASSERT in c2a5 -- skybox sentinel

2023-11-06 E325

• fix material asserts and inherit
• fixup -vkverboselogs
• changing textures on buttons, etc
• fix unpatched chrome surfaces brightness glitches

2023-11-03 E324

• add cvar for displaying only specified channel
• r_lightmap
• highlight all surfaces with random colors
• highlight selected surfaces -- decided to postpone
• massage shaders: consolidate all bindings explicitly
• skip sorting-by-texture when loading brush models (=> geometry count explosion; i.e. kusochki count will explode too)
• kusochki-vs-materials structures
• -vkverbose arg for turning all debug logs before detailed cvars are read

2023-11-02 E323

• lol meta: read and sort issues
• merge from upstream
• hevsuit glitches
• inverted normal map orientation

2023-10-31 E322

• load png blue noise files
• [-] translucent animated thing -> needs shader rework
• massage texture code
  • single return/goto cleanup
  • [-] pass args via structs? -> not necessary
  • [-] collapse texture uploading into a single function -> not necessary, they are different enough
• merge materials PR
• studio gibs translucency
• smoothing exclusion

2023-10-30 E321

• missing skybox
• explicitly free default textures; and complain about any leftovers
• use the new hash table in materials too, remove dummy textures
• why are there references to *unused
• restore blue noise
  • vk_texture_t blue_noise; 3d texture
  • separate binding similar to skybox in vk_rtx.c and shaders
  • patch shader function
  • load 64xpngs into a single big pic

2023-10-27 E320

• fix windows build
• track texture visibility for ref_api via flag and refcounts
• devmem assert, not all textures are destroyed in wagonchik
  • new material names+fixme => move to material hash table
  • preallocated default textures
• check urmom stats after a few different changelevels
  • COUNT(IS_DELETED)
  • clusters size histogram
• silence logs
  • "accessing empty texture"
  • "found existing texture"
• check mips

2023-10-26 E319

• fix pbr materials disappearing
• fix surface lights
• pbr/material refcount leaks
  • track texture visibility for ref_api
• handle existing image on texture upload
  • sanely recreate
  • reuse if possible
• case insensitive hash table

2023-10-24 E318

• use new hashmap for textures
  • use vk_texure array directly as open addressing hash table
    • Completely hide struct vk_texture
    • just try
    • texture indexes are no longer consecutive
  • blue noise texture breaks => make it a separate (3d) thing
  • index=0 is now valid
    • I. mark 0 as occupied to avoid allocating it
    • II. Increase all returned indexes by 1. Then dec it back wherever it is passed back
  • (SAD): cannot make builtin textures have stable indexes anymore

E313

Pre-next:

• validation crash

Next:

• KTX2 PR against upstream
• texture leaks
  • better texture storage
    • hash map
  • texture lifetimes/refcounts
  • texture deletion
    • mass (for single device wait idle)

Programmable render

• implicit dependency tracking. pass defines:
  • imports: list of things it needs
  • exports: list of things it produces. those get created and registered with this pass as a producer
• resource management refactoring:
  • register existing resources (tlas, buffers, temp images, ...) in their producers
  • resource automatic resolution: prducing, barriers, etc
  • resource destruction
• ? resource object: name, metadata(type, etc.), producer, status (ready, barriers, etc)

Multipass + Sampling

• better simple sampling
  • all triangles
  • area based on triangles
  • clipping?
  • can we pack polygon lights better? e.g.:
    • each light is strictly a triangle
    • index is offset into triangles
    • layout:
      • vec4(plane) // is it really needed? is early culling important? can we shove area into there too? e.g plane_n.xy,plane_d, area
      • vec4(v0xyz, e_r)
      • vec4(v1xyz, e_g)
      • vec4(v2xyz, e_b)

Next

• remove surface visibility cache
• rtx: rename point lights to lampochki
• rtx: rename emissive surface to surface lights
• rtx: dynamically sized light clusters Split into 2 buffers: struct LightCluster { uint16 offset, length; } uint8_t data[];

Planned

• improve nonuniformEXT usage: https://github.com/KhronosGroup/Vulkan-Samples/pull/243/files#diff-262568ff21d7a618c0069d6a4ddf78e715fe5326c71dd2f5cdf8fc8da929bc4eR31
• rtx: experiment with refraction index and "refraction roughness"
• emissive beams
• emissive particles/sprites
• issue: transparent brushes are too transparent (train ride)
  • (test_shaders_basic.bsp) shows that for brushes at least there are the following discrepancies with gl renderer:
    • traditional:
      • anything textured transparent is slightly darker in ref_vk
      • "Color" render mode should not sample texture at all and use just color
      • "Texture" looks mostly correct, but ~2x darker than it should be
      • "Glow" looks totally incorrect, it should be the same as "Texture" (as in ref_gl)
      • "Additive" is way too dark in ref_vk
  • rtx: - "Color" should use solid color instead of texture - "Color", "Texture", ("Glow"?) should be able to reflect and refract, likely not universally though, as they might be used for different intended effects in game. figure this out on case-by-case basis. maybe we could control it based on texture names and such. - "Additive" should just be emissive and not reflective/refractive
• rtx: filter things to render, e.g.: some sprites are there to fake bloom, we don't need to draw them in rtx mode
• possibly split vk_render into (a) rendering/pipeline, (b) buffer management/allocation, (c) render state
• studio models: fix lighting: should have white texture instead of lightmap OR we could write nearest surface lightmap coords to fake light
  • make it look correct lol
• studio model types:
  • normal
  • float
  • chrome
• rtx: sky light/emissive skybox:
  • consider baking it into a single (or a few localized) kusok that has one entry in light cluster
  • just ignore sky surfaces and treat not hitting anything as hitting sky. importance-sample by sun direction
  • pre-compute importance sampling direction by searching for ray-miss directions
• rtx: importance-sample sky light; there are sky surfaces that we can consider light sources
• cull water surfaces (see c3a2a)
• consider doing per-geometry rendermode: brushes can be built only once; late transparency depth sorting for vk render;
• rtx: too many emissive lights in c3a1b
• rtx: denoise
  • non local means ?
  • reprojection
  • SVG+
  • ...
• rtx: bake light visibility in compute shader
• make 2nd commad buffer for resource upload
• ❌ bad condition for temp vs map-permanent buffer error message
• fix brush blending
• sprite depth offset
• fix incorrect viewport sprite culling
• improve g_camera handling; trace SetViewPass vs RenderScene ...
• studio model lighting
• ❌ move all consts to vk_const
• decals
• lightmap dynamic styles
• fog
• studio models survive NewMap; need to compactify buffers after removing all brushes
• sometimes it gets very slow (1fps) when ran under lldb (only on stream?)
• rtx: non-realtime unbiased mode: make "ground truth" screenshots that take 1e5 samples per pixels and seconds to produce. what for: semi-interactive material tuning, comparison w/ denoise, etc.

Someday

• more than one lightmap texture. E.g. sponza ends up having 3 lightmaps
• better 2d renderer: fill DRAWQUAD(texture, color, ...) command into storage buffer instead of 4 vertices
• brush geometry is not watertight
• collect render_draw_t w/o submitting them to cmdbuf, then sort by render_mode, trans depth, and other parameters, trying to batch as much stuff as possible; only then submit

2021-02-06

• alpha test
• compare w/ gl R_SetRendeMode
  • raster state
  • color constants
• culling
• shaders s/map/brush/
• pipeline cache
• swapchain getting stale
• HUD sprites
• issue: lightmap sometimes gets corrupted on map load

2021-02-08

• move entity rendering-enumeration into vk_scene

2021-02-10

• refactor brush into brushes and separate rendering/buffer management
• animated textures (accept PR)

2021-02-13

• move pipelines from brush to render
• render temp buffer api
• draw studio models somehow
• studio models vk debug markers
• studio models white texture as lightmap
• studio models fixes

2021-02-15

• weapon models -- viewmodel
• coalesce studio model draw calls
• initual sprite support

2021-02-17

• draw some beams

2021-02-20

• refactor vk_render interface:
  • move uniform_data_t to global render state ~inside render_draw_t, remove any mentions of uniform/slots from api; alt: global render state?~
  • rename RenderDraw to SubmitDraw
  • ~add debug label to render_draw_t?;~ alt: VK_RenderDebugNameBegin/End
  • perform 3d rendering on corresponding refapi calls, not endframe
• fix sprite blending

2021-02-22

• RTX: load extensions with -rtx arg
• vk_render: buffer-alloc-centric upload and draw api

2021-03-06

• (RTX; common) Staging vs on-GPU buffers
• rtx: BLAS construction on buffer unlock
• rtx: ray trace compute shader
• dlight test

2021-03-08

• studio models normals
• rtx: geometry indexing

2021-03-10

• rtx: dlights
• rtx: dlight shadows
• rtx: dlight soft shadows

2021-03-13

• rtx: blend normals according to barycentrics
• rtx: (debug/dev) shader reload
• rtx: make projection matrix independent render global/current/static state
• rtx: model matrices
• rtx: light entities -- still not enough to enlight maps :(
• rtx: path tracing

2021-03-15

• rtx: control bounces with cvars
• rtx: device-local buffers -- doesn't affect perf noticeably :(
• rtx: emissive materials
  • rtx: emissive textures
  • rtx: emissive beams

2021-03-17..20

• rtx: lower resolution framebuffer + upscale
• rtx: importance sample emissive surface
• rtx: remove entnity-parsed lights
• rtx: naive temporal denoise: mix with previous frame

2021-03-22

• rtx: traverse bsp for science!

2021-03-28

• bake s/d-lights visibility data into bsp leaves

2021-04-06..08

• persistent models
  • load brushes into render model
  • destroy brushes when time comes (when?)
  • rasterize models in renderer

2021-04-09

• rtx: build AS for model
• rtx: include pre-built models in TLAS

2021-04-10

• rtx: fix tlas rebuild
• rtx: upload kusochki metadata w/ leaves
• rtx: add fps
  • rtx: don't group brush draws by texture
  • better AS structure (fewer blases, etc)

2021-04-11

• vscode build and debug

2021-04-12

• rtx: fix surface-kusok index mismatch
• rtx: try to use light visibility data
  • too few slots for light sources
  • some areas have too many naively visible lights
• rtx: fix light shadow artefacts

2021-04-13

• rtx: "toilet error": attempting to get AS device address crashes the driver
• rtx: fix blas destruction on exit
• rtx: sometimes we get uninitialized models

2021-04-14..16

• rtx: grid-based light clusters

2021-04-17

• rtx: read rad file data

2021-04-19

• rtx: light intensity-based light clusters visibility
• rtx: check multiple variants of texture name (wad and non-wad)
• rtx: rad liquids/xeno/... textures

2021-04-22

• rtx: fix backlight glitch
• rtx: textures

2021-04-24, E86

• rtx: restore studio models

2021-05-01, E89

• make a wrapper for descriptor sets/layouts

2021-05-03, E90

• make map/frame lifetime aware allocator and use it everywhere: render, rtx buffers, etc

2021-05-08, E92

• rtx: weird purple bbox-like glitches on dynamic geometry (tlas vs blas memory corruption/aliasing)
• rtx: some studio models have glitchy geometry

2021-05-10, E93

• rtx: don't recreate tlas each frame
• rtx: dynamic models AS caching

2021-05-..-17, E93, E94

• rtx: improve AS lifetime/management; i.e. pre-cache them, etc
• add debug names to all of the buffers

2021-05-22, E97

• add nvidia aftermath sdk

2021-05-24, E98

• rtx: simplify AS tracking

2021-05-26, E99

• rtx: fix device lost after map load

2021-05-28, E100

• rtx: build acceleration structures in a single queue/cmdbuf

2021-06-05, E103

• rtx: dynamic surface lights / dynamic light clusters
• rtx: animated textures
• rtx: attenuate surface lights by normal

2021-06-07, E104..

• fix CI for vulkan branch

2021-06-09..12, E105..106

• c3a2a: no water surfaces in vk (transparent in gl: *45,*24,*19-21)
• water surfaces

2021-06-14, E107

• rtx: optimize water normals. now they're very slow because we R/W gpu mem? yes
• cull bottom water surfaces (they're PLANE_Z looking down)
• fix water normals

2021-06-23, E109

• rtx: ray tracing shaders specialization, e.g. for light clusters constants
• rtx: restore dynamic stuff like particles, beams, etc
• rtx: c3a1b: assert model->size >= build_size.accelerationStructureSize failed at vk_rtx.c:347

2021-07-17, E110..120

• rtx: ray tracing pipeline
• rtx: fix rendering on AMD
• rtx: split models into a separate module
• rtx: alpha test

2021-07-31, E121

• rtx: alpha blending -- did a PoC

2021-08-02..04, E122-123

• mipmaps
• rtx: better random

2021-08-07, E124

• anisotropic texture sampling
• studio model lighting prep
  • copy over R_LightVec from GL renderer
  • add per-vertex color attribute
  • support per-vertex colors
  • disable lightmaps, or use white texture for it instead

2021-08-11, E125

• simplify buffer api: do alloc+lock as a single op

2021-08-15, E126

• restore render debug labels
• restore draw call concatenation; brush geoms are generated in a way that makes concatenating them impossible

2021-08-16, E127

• better device enumeration

2021-08-18, E128

• rtx: fix maxVertex for brushes

2021-08-22, E129

• fix depth test for glow render mode
• screenshots

2021-08-26, E131

• rtx: material flags for kusochki

2021-09-01, E132

• rtx: ingest brdfs from ray tracing gems 2
• rtx: directly select a triangle for light sampling

2021-09-04, E133

• rtx: different sbts for opaque and alpha mask
• include common headers with struct definitions from both shaders and c code

2021-09-06, E134

• rtx: pass alpha for transparency
• rtx: remove additive/refractive flags in favor or probability of ray continuing further instead of bouncing off
• make a list of all possible materials, categorize them and figure out what to do

E149

• rtx: remove sun

• rtx: point lights:

  • static lights
    • intensity "fix"
  • dlights
  • elights
  • intensity fix for d/elights?
  • point light clusters
  • bsp:
    • leaf culling
    • pvs

• rtx: better light culling: normal, bsp visibility, (~light volumes and intensity, sort by intensity, etc~)

• rtx: cluster dlights

2021-10-24 E155

• rtx: static lights
  • point lights
  • surface lights

2021-10-26 E156

• enable entity-parsed lights by lightstyles

2021-12-21 DONE SOMEWHEN

• rtx: dynamic rtx/non-rtx switching breaks dynamic models (haven't seen this in a while)
• run under asan
• rtx: map name to rad files mapping
• rtx: live rad file reloading (or other solution for tuning lights)
• rtx: move entity parsing to its own module
• rtx: configuration that includes texture name -> pbr params mapping, etc. Global, per-map, ...
• rtx: simple convolution denoise (bilateral?)
• rtx: cull light sources (dlights and light textures) using bsp
• [-] crash in PM_RecursiveHullCheck. havent seen this in a while
• rtx: remove lbsp

2022-09-17 E207 Parallel frames

• allocate for N frames:

  • geometries
  • rt models
    • kusochki
      • same ring buffer alloc as for geometries
        • extract as a unit
    • tlas geom --//--
  • [-] lights
    • make metadata buffer in lights
    • [-] join lights grid+meta into a single buffer => pipeline loading issues
    • put lights data into a cpu-side vk buffer
    • [-] sync+barrier upload => TOO BIG AND TOO SLOW, need to e.g. track dirty regions, compactify stuff (many clusters are the same), etc

• scratch buffer:

  • should be fine (assuming intra-cmdbuf sync), contents lifetime is single frame only

• accels_buffer:

  • lifetime: multiple frames; dynamic: some b/tlases get rebuilt every frame
  • opt 1: double buffering
  • opt 2: intra-cmdbuf sync (can't write unless previous frame is done)

• uniform_buffer:

  • lifetime: single frame

• tlas_geom_buffer:

  • similar to scratch_buffer
  • BUT: filled on CPU, so it's not properly synchronsized
  • fix: upload using staging?
    • double/ring buffering

• E213:

  • parse binding types
  • remove types from resources FIXME

• E214: ~tentative~

  • integrate sebastian into waf

• E215:

  • serialize binding image format

2022-11-26 E216 rake yuri

- [x] validate meatpipe image formats
- [x] begin Rake Yuri migration
	- [x] direct lights

2023-01-21 E217-E221

• meatpipe resource tracking
  • name -> index mapping
  • create images on meatpipe load
  • automatic resource creation
  • serialize all resources with in/out and formats for images
  • create resources on demand
• parse spirv -> get bindings with names
  • spirv needs to be compiled with -g, otherwise there are no OpName entries. Need a custom strip util that strips the rest?
    • unnamed uniform blocks are uncomfortable to parse.
• passes "export" their bindings as detailed resource descriptions:
  • images: name, r/w, format, resolution (? not found in spv, needs to be externally supplied)
    • [-] buffers: name, r/w, size, type name (?) -- can't really do, too hard for now
• name -> index resolver (hashmap kekw)
• automatic creation of resources
  • images
  • [-] buffers -- no immediate need for that

2023-01-22 E222

• refcount meatpipe created images
• rake yuri primary ray

2023-01-28 E223

• previous frame resources reference - specification: - I: prev_ -> resource flag + pair index - II: new section in json - internals: - I: create a new image for prev_, track its source; swap them each frame Result is meh: too much indirection, hard to follow, many things need manual fragile updates. - II: create tightly coupled image pair[2], read from [frame%2] write to [frame%2+1] - III: like (I) but with more general resource management: i.e. resource object for prev_ points to its source

2023-01-28-02-08 E224-229

• light_grid_buffer (+ small lights_buffer):
  • lifetime: single frame
  • BUT: populated by CPU, needs sync; can't just ring-buffer it
  • fixes: double-buffering?
    • staging + sync upload? staging needs to be huge or done in chunks. also, cpu needs to wait on staging upload
    • 2x size + wait: won't fit into device-local-host-visible mem
    • decrease size first?
• additive transparency
• bounces
• skybox shadows
• [-] rtx: shrink payload between shaders
• rtx: split ray tracing into modules: pipeline mgmt, buffer mgmt
• nvnsight into buffer memory and stuff
• multiple frames in flight (#nd cmdbuf, ...)
• embed shaders into binary
• verify resources lifetime: make sure we don't leak and delete all textures, brushes, models, etc between maps
• custom allocator for vulkan
• rtx: better mip lods: there's a weird math that operates on fov degrees (not radians) that we copypasted from ray tracing gems 2 chapter 7. When the book is available, get through the math and figure this out.
• render skybox
• better flashlight: spotlight instead of dlight point
• rtx: add fps: rasterize into G-buffer, and only then compute lighting with rtx

Done somewhen

• create water surfaces once in vk_brush
• loading to the same map breaks geometry
• (helps with RTX?) unified rendering (brush/studio models/...), each model is instance, instance data is read from storage buffers, gives info about vertex format, texture bindings, etc; which are read from another set of storage buffers, ..
• waf shader build step -- get from upstream

Collected on 2024-01-18

• what if new meatpipe has different image format for a creatable image?
• rtx: light styles: need static lights data, not clear how and what to do
• more beams types
• more particle types
• sane texture memory management: do not allocate VKDeviceMemory for every texture
• rtx: transparency layering issue, possible approaches:
  • trace a special transparent-only ray separately from opaque. This can at least be used to remove black texture areas
• rtx: better memory handling
  • robust tracking of memory hierarchies: global/static, map, frame
  • or just do a generic allocator with compaction?
• rtx: coalesce all these buffers
• rtx: entity lights
• rtx: do not rebuild static studio models (most of them). BLAS building takes most of the frame time (~12ms where ray tracing itself is just 3ms)
• studio models: pre-compute buffer sizes and allocate them at once
• dlight for flashlight seems to be broken
• fix sprite blending; there are commented out functions that we really need (see tunnel before the helicopter in the very beginning)
• fix projection matrix differences w/ gl render
• what is GL_Backend*/GL_RenderFrame ???
• particles
• optimize perf: cmdbuf managements and semaphores, upload to gpu, ...
• rtx: studio models should not pre-transform vertices with modelView matrix
• start building command buffers in beginframe
• cleanup unused stuff in vk_studio.c
• stats
• [-] auto-atlas lots of smol textures: most of model texture are tiny (64x64 or less), can we not rebind them all the time? alt: bindless texture array
• can we also try to coalesce sprite draw calls?