set the channel description: Channel to discuss design & implementation to improve the documentation experience within DataHub

Hey everyone, as mentioned in the call we did a quick analysis on existing text editor frameworks that would power-up and improve the user experience. We’ve shortlisted 2 solutions: tiptap.dev and lexical.dev. Although lexical.dev’s demo seems to provide a lot more functionalities than tiptap.dev, a wide list of functionalities are not finalised yet and documentation is lacking. Furthermore, it’s in early development so maintenance in the future may pose a huge issue. We’re leaning more towards using tiptap.dev instead. Do take a look and let us know what you think. https://docs.google.com/document/d/1JYQSgE-mLgyhq8FApv-PQaiopEm9eTI50C4FUY9jqwM/edit?usp=sharing

Hi folks! Here’s some additional context about why we created this channel. @big-carpet-38439 and I met with @brave-insurance-80044 (Grab) & @boundless-student-48844 (Grab) yesterday to discuss some proposed improvements to DataHub’s documentation functionality. Separately, we’ve been having conversations with folks from GResearch to see if there are ways we can collaborate on improvements to the UI; hoping this is a great first initiative! Here’s my understanding of the scope of this design/discovery work: Use Cases/User Studies • Provide more robust documentation functionality within DataHub to help end-users capture & share institutional knowledge ◦ In scope: All entity-level descriptions, column descriptions? • Support tagging/associating DataHub entities and DataHub Users within the documentation itself (similar to GSuite or other Wiki functionality to support inline lookup/linking) • Support version control for changes made to documentation; easily view who made which change; toggle between historical versions • Upload files/images into documentation • Move away from markdown-first editors to remove friction for users not well-versed in .md formatting Out of Scope for now: • Creating a new documentation-specific entity type within DataHub; we want to see how much of an impact we can have without further complicating the underlying entity model (Glossary Terms, Domains, etc.) Discovery/Decisions to be made: • What is the most stable editor for us to implement? — Please see @brave-insurance-80044 doc above!! • If we change editors, what is the impact on existing markdown docs? What is the migration strategy? • If we support @-mentions, what is the notification story within OSS DataHub? • If we support file/image upload, will this require net-new file storage functionality within DatHub? • What is the user experience if documentation is pushed programmatically during ingestion? Does it replace UI-generated documentation?

-- Grab Team: My hope is we can find ways to have this group collaborate on discovery, design, & implementation, without blocking your team’s internal objectives/OKRs for the remainder of the year. If at any point we’re putting you all at risk for missing a deadline, please loop me in so we can resolve blockers asap!

Core DataHub Team: My expectation is that we will be helping with discovery and implementation of this initiative; we’ll need to figure out timeline/scope/prioritization/etc, but this is a great opportunity for us to take steps towards improved collaboration functionality while teaming up with our amazing community members teamwork

<!channel> Hello, all! I want to make sure we’re able to maintain momentum with this effort, so I am going to schedule some recurring reminders/check-ins for this group, targeting once every 2wks in Sept and Oct. If you have bandwidth to contribute to this effort, please add your global timezone in the thread and general availability for Zoom checkins (ie. morning/afternoon/evening)… I’m not sure we’ll be able to find a time that works for everyone across timezones, so we might have to get crafty with async checkins!

We should start by defining what the end state of documentation editor should be. What pieces are mission critical vs nice to have? Some ideas are... • Multi-player documentation (syncing docs across live users) • Full screen editor (vs current inline editor) • HTML editor (vs current markdown editor) • Keyboard shortcuts etc

Following up here - I scheduled the first session for next Tuesday. @brave-insurance-80044 @boundless-student-48844 please feel free to add anyone from your side. @bulky-soccer-26729 is going to be the main point of contact from the Core DataHub Team to provide technical guidance as we go along, and I’ll be on point to help keep us all on track

Excited to kick this off further 🙂

Same! Tomorrow’s the dayyyyy

Happy Monday team! Reviving this channel with some draft Figma Designs. Will share more soon 😄

Hi folks! I created a Polly poll to send out to all members of our Slack instance via DM to learn more about removing Markdown support. It’s scheduled to go out on Tuesday, October 18th at 12:00pm EST (this is following Polly’s guidance about optimal time to send for higher response rates). It will stay open for 10 days, and a reminder will be sent out on the 7th day. Attaching screenshots in 🧵 of the poll

@brave-insurance-80044 @boundless-student-48844 @bulky-soccer-26729 Polly results will be flowing into this Google Sheet if you want to keep an eye on it (I’ve hidden all identifying/contact information); it will close on October 28th. We have 52 responses so far; the tl;dr is about 25% of folks have a requirement on Markdown

Hey @bulky-soccer-26729 @boundless-student-48844 @brave-insurance-80044! We received 93 responses to the survey; let’s plan on reviewing high-level outcomes during our session tomorrow!

Hello, I'm afraid we were a bit late getting the poll in front of our DataHub users, so we missed the deadline. However, I've shared it with them now and there are a couple of concerns about dropping markdown support. Would it be OK to share here?

Hi. I was invited here by @big-carpet-38439 as we're trying to introduce DataHub into our company and we have some specific requirements for structuring, editing and displaying documentation for datasets. We're in the process of more formally gathering these but is this a good place to talk about them? Sorry if we're late to the party!

Hey folks! First off, the Grab team would like to thank DataHub for sending out the Polly poll to better understand the community’s usage on the documentation feature. And also thanks to everyone who joined this channel to share their thoughts! With that, I’d like to provide a summary of what we’ve been working on so far and to share some updates. Also added some screenshots too! We’re building this within our own fork of DataHub so stylings will differ slightly. • We’ll not be dropping Markdown support. Markdown is here to stay and will continue to be format for storing these documentation. • New features provided ◦ @-mentions - users will be able to tag their entities in their documentation. These tags will be stored in as

[@<entity_name>](<entity_urn>)

in Markdown (e.g.

[@SampleHdfsDataset](urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD))

). If you’re using other Markdown editors, these should most likely appear as the tagged name only like:

@SampleHdfsDataset

. • UX improvements ◦ WYSIWYG Editor - users will no longer be required to write in Markdown but Markdown shortcuts will still be supported ◦ Tables would show more user-friendly controls to add/remove rows and columns ◦ Users can add images and links using a proper modal with a form instead For existing documentations that are using GitHub Flavoured Markdown (GFM), rest assured that the new editor will still be able to render your content. If you have any suggestions, please feel free to let us know too!

This looks incredible

Having a conversation with G-Research folks now. A few concrete requirements they had requested - a concrete Table of Contents

(Navigation to a section quickly)

@brave-insurance-80044 Is that something you guys are also interested in?

Hi @brave-insurance-80044 & @boundless-student-48844! Wanted to introduce you to Lucca - he is interested in contributing support for Mermaid diagrams via markdown using mermaid.js (feature request here). @numerous-eve-42142 - Amanda & Harvey are on the Grab team and have been working on a contribution to improve our documentation functionality (targeting to contribute it by the end of the year). I’m not sure how much overlap there is between the two workstreams, but wanted to make sure you were connected just in case!

Hi Datahub team, any advice regarding self-hosted image upload solution? https://datahubspace.slack.com/archives/C02FKQAGRG9/p1664606921413959 I have check that Restli API support attachment? @mammoth-bear-12532 possible to give some guidance regarding this? Or Datahub team have any intention to support this?

archived the channel