> Below is a dump of notes that should help you get started: @User This makes sense, thanks. File extension should be a good enough way of identifying the file type. If I wanted to hook some kind of export process in to processing the link, I guess it would be good to expose this as an event that other VS Code extensions can hook into? I'll need have a play with the split across the core and VSC extensions and see what's involved, because I'm not sure what should be "Dendron core" and what is VSC specific

the way we've split it up - anything that depends on vscode apis goes into the plugin. anything that isn't should go outside. ideally, dendron's core functionality should be editor agnostic (there are some folks that are investigating sublime and vim extensions for it). did you have something specific in mind when you say expose as an 'event'?

@User in this case, if my Draw.io diagram is in source form, I need to export it. Instead of tying Dendron directly into that I was thinking of raising an event with the ref in, letting another extension handle it. It might be more appropriate to have those extensions register a handler function that can return a promise, though?

i guess i'm still a little confused what you meant when you say event. are you talking about exporting a library with an

EventEmitter

?

Sure, some means of notifying an “exporter” that a file of its type is about to be embedded, allowing it to be exported. I think it should be possible to declare a handler for each ref type that emits the markup for the preview. Draw.io documents might need to be exported to SVG or a raster, and since multiple pages can be embedded in a given document I would want to pick the specified page.

The more I think about the more I think I want a Promise instead, as it’s allow the handler to provide the markup for embedding that reference rather than just linking to some external resource?

@User so the way the preview works is that there's a giant

parse

loop that scans the plaintext and converts it to html. when it encounters a ref, we have a ref handler that will read the file and convert it. for refs, this is all done within the

parseNoteRefs

plugin: https://github.com/dendronhq/dendron-template/blob/master/vault/dendron.dev.design.engine.md#parsing-note-references if you come up with a good way of doing events/handlers besides just registering all ref handlers at startup, feel free to come up with a proposal 🙂

after diving into LSP, what are your current thought on the subject, @User ?

@User wrote up some thoughts here. feedback appreciated! https://github.com/dendronhq/dendron-template/blob/master/vault/project.n.2020.lsp.md#L1:L1

@User Good initial stab! A few comments: - From the document, it's not quite obvious what the role of the engine-client is. I might have misunderstood it in the comments below. - I personally would partition this maybe slightly differently: 1. Core / Engine, the "business logic" for most of the functionalities like lookups, diagnostics, publishing etc. 2. LSP server, delegates calls to core 3. VSCode extension, calls LSP server and manages the UI. In some cases might call core directly, but the execution will then happen in VS Code process (see below). 4. CLI, calls core directly - To get the lookup results into extension, I believe server extension communication can be extended with custom JSON RPC messages (thus: extension asks completion -> server asks it from core -> core constructs that from its index, returns to server -> server returns result to extension) - IIRC, file watching is done by the VS Code, and it will notify the LSP server whenever any file relevant for the extension is changed. Typically every change would cause index refresh + sending status + diagnostics from server to extension (probably with some throttling). - IIRC, the watched files are set in the extension's package.json - Things like creating daily notes, I would delegate via extension -> core, so that CLI can then do similar thing via CLI -> core (thus logic for it would only reside in core)

@User got it. so instead of

plugin/cli -> engine -> lsp

, you're proposing

plugin/cli -> lsp -> core

. in this case,

lsp

would basically serve as an

api gateway

of sorts, abstracting the implementation details of the engine behind the lsp > To get the lookup results into extension, I believe server extension communication can be extended with custom JSON RPC messages (thus: extension asks completion -> server asks it from core -> core constructs that from its index, returns to server -> server returns result to extension) this is where I need to get more clarity on. wasn't obvious whether LSP supported custom messages or if we had to overload one of the existing functionalities > - IIRC, file watching is done by the VS Code, and it will notify the LSP server whenever any file relevant for the extension is changed. Typically every change would cause index refresh + sending status + diagnostics from server to extension (probably with some throttling). i'm up in the air about this. if we want the lsp/core to be vscode independent, it would be easier to delegate file watching to the server. plus it'll use the vscode fs watcher interface present day so functionally, wouldn't be a big change either way. are there advantages for doing file watching inside vscode isntead? > - Things like creating daily notes, I would delegate via extension -> core, so that CLI can then do similar thing via CLI -> core (thus logic for it would only reside in core) i like this. core would continue to index, lsp is just the messaging layer that helps decouple the engine/core from its js implementation.

> you're proposing plugin/cli -> lsp -> core Almost, to be exact it would be (plugin -> ((server -> core) / core)) and (cli -> core) > wasn't obvious whether LSP supported custom messages In primitive level, it's JSON RPC so I guess pretty much anything goes. The typescript implementation provided by MS has some generic handlers which probably can be used for this (haven't tried though). See e.g. https://github.com/microsoft/vscode-languageserver-node/blob/60a5a7825e6f54f57917091f394fd8db7d1724bc/server/src/common/server.ts#L1006 > are there advantages for doing file watching inside vscode isntead? Yes, see https://microsoft.github.io/language-server-protocol/specification#workspace_didChangeWatchedFiles

> Almost, to be exact it would be (plugin -> ((server -> core) / core)) and (cli -> core) got it. yeah, this feels right > In primitive level, it's JSON RPC so I guess pretty much anything goes that's fair. regardless, it wouldn't be too hard to pass custom messages one way or another > are there advantages for doing file watching inside vscode isntead? that's fair. i guess if we wanted to support different editors in the future, as long as we had a general

onFileChange

handler, the file watching can be client specific

Yeah, I believe most of the editors would probably implement the lsp like recommended, so the server would work. Its "just" getting the plugin (and the related UIs) implemented into various environments.

what might turn out to be somewhat difficult is to keep core clean of microsoft's vscode-languageserver types. In my trial these types started easily to spread around the codebase, keeping everything clean in core would probably require plenty of duplication of types so it's probably not worth it - at least in the first iteration.

yeah, i agree. first iteration will be best effort to keep the core clean, with a serious attempt when we or someone else gets around to doing a second editor implementation

@User yesterday I promised to summarize my point of view to the "dot-based hieararchies" vs. "folders" discussion. Here it goes:

. # Dotted filenames - note renames are simpler, no need to create / delete folders - note containers can be created with a single artifact (e.g. foo.bar.md is the container for foo.bar vs. foo/bar.md + foo/bar/) - note stubs can be created without intermediate folders # Folder Hierarchies - familiarity - dendron file structure looks like any other part of the file system. Less to learn, faster to adopt - you can place any files (e.g. pdfs, xlsx or draw.io diagrams) into the folders without renaming them (and reference them with the same mechanics) - easier to save files into the right place of the structure without the assistance of vscode (e.g. normal drag-and-drop works) - natural "zoom" functionality into the structure (just open the folder) - filenames look cleaner in vscode (in several views, like explorer, file tabs, git etc) as well as in normal OS utilities - large amount of files in the same folder makes normal file system tools less useful (OS file explorer, commands like ls or tree, potential choking of operations) - less implementation needed for dendron (e.g. no need for dendron tree view since normal file explorer works) - other vscode extensions rely on file structures for cleaner views (e.g. TODO tree). Not all views can be re-implemented for dendron - normal file system permissions could be used to limit visibility to certain sub-hierarchies - various sub-parts of the hierarchy can be brought into the same workspace via symlinks - Is compliant with enterprise setting, many companies just wouldn't adopt dot-based hierarcies for their knowledge-base (it's "too weird", "unproven" etc.) .

probably not a complete list, but it's a start

...further comments welcome

Using folders would also solve the sharing between private and enterprise stuff by the way

There might also be differences related to vscode search across files. With dotted filenames you can search for

foo*bar

but the same query won't turn up a

foo/bar.md

. (meaning files to include/exclude options)

@User do you mean the link syntax? If you mean including / excluding certain subhierarchies, I believe this should work

I mean this search) and "files to include/exclude" lines. Not too familiar with it, but seems to me that things work somewhat differently for dotted filenames and folder structures here.

Thanks for writing this up :) I also added this to our dev documents here to track the salient points of this dicussion: https://github.com/dendronhq/dendron-template/blob/master/vault/dendron.dev.design.files-vs-folders.md#L25:L25 I agree that moving to folders will simplify integrating with existing tool chain around files/folders. I think there are workarounds we can implement to smooth over the differences but they will be workarounds at the end of the day. Moving from files to folders will be a significant development effort without any immediate change in functionality so I'm just wary about undertaking this until we have more resources or a significant portion of the community wants this change. As part of the deliverables for this month, I want to make the dendron roadmap public and open it up for voting. @User , can you create an issue for this as well and link it to the design docs? Maybe we can do a vscode thing where if enough people vote for it, then we'll take it on. Some additional questions: - if we do switch to folders, is it worth keeping the

.

syntax at all? have it as a legacy fallback and slowly deprecate? - if we switch to folders, do we also switch the lookup syntax to use '/' instead of '.'? @User as part of the search, I believe you can achieve the same using '/' instead of '*'

well not if my note is

foo.x.y.z.bar

and I totally forgot about x, y and z))

the following syntax would work:

foo/**/*.md

**/*

=

all files recursively under this directory

on an unrelated note, a bunch of people have brought up doing roam like queries inside dendron. created an issue for it here. it's actually quite natural to implement this as an extension of note references. would love to hear y'all's thoughts on this https://github.com/dendronhq/dendron/issues/209

what's an

area

?