My Journey with LLVM (GSoC'20 Phase 1)

It has been one month since my proposal gets accepted by GSoC'20. I learned a lot and had a wonderful time. Besides, we’ve made some progress towards our goal. Hence, it’s a good time to review what I’ve done and what I’ve learned in the first coding period.

The Project#

In LLVM, we use yaml2obj to handcraft simple binaries of various formats in YAML, e.g., ELF, Mach-O, COFF, etc. My project is to add DWARF support to yaml2obj which hopefully makes it easier for people to handcraft debug sections in those kinds of binaries. This project is supervised by James Henderson.

The Progress#

We’ve already ported existing DWARF implementation to yaml2elf as planned. People are able to handcraft DWARF sections at a low level. I have to admit that the current implementation of DWARF sections is hard to use since we have to specify nearly every field of those sections, e.g., the length, the version, the address or offset of the associated DWARF section, etc. That’s because those sections are isolated in the current implementation and DWARFYAML lacks a strategy to make those sections get interlinked properly. This is what we are going to address and I believe it will be improved in the future. We also have a spreadsheet to record the progress against the expected timeline.

The Implementation Status#

The supported DWARF sections’ syntax and known issues are listed below. I’m not going to resolve all of the issues since some DWARF sections are deprecated in DWARFv5 spec and rarely used.

Note: The fields quoted by "[[]]" are optional.

Syntax	Known Issues/Possible Improvements
	* Doesn't support emitting multiple abbrev tables. D83116
	* `yaml2macho` doesn't support emitting the `.debug_addr` section. * `dwarf2yaml` doesn't support parsing the `.debug_addr` section.
	* The `Length`, `AddrSize` and `SegSize` fields should be optional. * Rename `CuOffset` to `DebugInfoOffset`. * Rename `AddrSize` to `AddressSize`. * Rename `SegSize` to `SegmentSelectorSize`.
	* Rename `AbbrOffset` to `DebugAbbrevOffset`. * Rename `AddrSize` to `AddressSize`. * Rename `AbbrCode` to `AbbrevCode` or `Code`.
	* The DWARFv5 `.debug_line` section isn't tested.
	* Doesn’t support emitting multiple pub tables. * Replace `Length` with `Format` and `Length`.

Accomplishments#

I’m very happy that I’m roughly able to reach the goal of the first period. During the first coding period, I learned about how the debug information is represented at a lower level in object files and how to process errors in the LLVM library. I’m also able to dig into some related core libraries, such as DebugInfo, CodeGen, and so on.

Areas in Need of Improvements#

However, there are still some areas that I didn’t do well. When I was working on porting DWARF support to yaml2elf, I found that some DWARF sections were not well-formatted, e.g., the .debug_pub* sections don’t support emitting multiple pub tables, the .debug_abbrev section doesn’t support emitting multiple abbreviation tables, the .debug_pub* and .debug_abbrev sections lack terminating entries, etc. I used to port them to yaml2elf first and then try to fix the issue. However, it’s not the right approach! I should have fixed the issue first and then ported the section to yaml2elf so that I don’t have to update the test cases in many places and this prevents ill-formed test cases from spreading everywhere.

Besides, if I had made elf2yaml support converting DWARF sections back to YAML, my life would be easier. After porting some sections to yaml2elf, I realize that it’s good for us to have a tool that is able to convert DWARF sections back so that I don’t have to handcraft too many sections.

Acknowledgements#

I would love to express my sincere gratitude to James Henderson for mentoring me during this project, and to folks for reviewing my patches and giving many useful suggestions in my proposal!

Accepted Patches#

In case these patches are useful for evaluation.

D82435 [DWARFYAML][debug_gnu_*] Add the missing context D82933 [DWARFYAML][debug_abbrev] Emit 0 byte for terminating abbreviations. D82622 [DWARFYAML][debug_info] Replace 'InitialLength' with D82367 [ObjectYAML][ELF] Add support for emitting the .debug_gnu_pubnames/pubtypes D82630 [ObjectYAML][DWARF] Collect diagnostic message when YAMLParser fails. D82296 [ObjectYAML][ELF] Add support for emitting the .debug_pubnames section. D82621 [DWARFYAML][debug_info] Teach yaml2obj emit correct DWARF64 unit header. D82351 [ObjectYAML][DWARF] Remove unused context. NFC. D82347 [ObjectYAML][ELF] Add support for emitting the .debug_pubtypes section. D82275 [DWARFYAML][debug_info] Add support for error handling. D82173 [DWARFYAML][debug_info] Use 'AbbrCode' to index the abbreviation. D82139 [DWARFYAML][debug_info] Fix array index out of bounds error. D82073 [ObjectYAML][ELF] Add support for emitting the .debug_info section. D81826 [DWARFYAML][debug_abbrev] Make the abbreviation code optional. D81820 [ObjectYAML][ELF] Add support for emitting the .debug_abbrev section. D81915 [ObjectYAML][DWARF] Let writeVariableSizedInteger() return Error. D81541 [ObjectYAML][DWARF] Implement the .debug_addr section. D81709 [ObjectYAML][DWARF] Let the target address size be inferred from FileHeader. D81529 [ObjectYAML][test] Use a single test file to test the empty 'DWARF' D80722 [ObjectYAML][DWARF] Make the PubSection optional. D81220 [DWARFYAML][debug_ranges] Make the "Offset" field optional. D81528 [DWARFYAML] Add support for emitting DWARF64 .debug_aranges section. D81450 [ObjectYAML][ELF] Add support for emitting the .debug_line section. D81357 [DWARFYAML][debug_ranges] Emit an error message for invalid offset. D81356 [ObjectYAML] Add support for error handling in DWARFYAML. NFC. D80203 [ObjectYAML][DWARF] Add DWARF entry in ELFYAML. D80862 [ObjectYAML][test] Address comments in D80203. D81217 [ObjectYAML][DWARF] Support emitting .debug_ranges section in ELFYAML. D81063 [DWARFYAML][debug_aranges] Replace InitialLength with Format and Length. D81051 [ObjectYAML][ELF] Let the endianness of DWARF sections be inferred D80972 [ObjectYAML][DWARF] Support emitting the .debug_aranges section in ELFYAML. D80535 [ObjectYAML][MachO] Add error handling in MachOEmitter.

'Format' and 'Length'.

sections.

entry.

from FileHeader.