catfood/LogSeq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

OSTEP Chapter 41

Browse Source
This commit is contained in:
ridethepig 2023-04-16 00:43:33 +08:00
parent e2e7ebf256
commit 1b2c362cdd
6 changed files with 1687 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1313
assets/ostep_1681115599584_0.edn
View File

File diff suppressed because it is too large Load Diff

1
journals/2023_04_14.md Normal file
View File

@ -0,0 +1 @@
- 终于把排名开下来了。

1
journals/2023_04_15.md Normal file
View File

@ -0,0 +1 @@
- 英文版简历绝赞制作中

1
logseq/bak/logseq/config/2023-04-13T15_47_53.912Z.Desktop.edn → logseq/bak/logseq/config/2023-04-15T15_41_51.461Z.Desktop.edn
View File

@ -343,6 +343,7 @@
;; ;use Percent-encoding for other invalid characters
:file/name-format :triple-lowbar
:ui/show-brackets? false
:feature/enable-timetracking? false
;; specify the format of the filename for journal files
;; :journal/file-name-format "yyyy_MM_dd"

1
logseq/bak/logseq/config/2023-04-13T11_23_48.892Z.Desktop.edn → logseq/bak/logseq/config/2023-04-15T15_42_07.581Z.Desktop.edn
View File

@ -343,6 +343,7 @@
;; ;use Percent-encoding for other invalid characters
:file/name-format :triple-lowbar
:ui/show-brackets? true
:feature/enable-timetracking? false
;; specify the format of the filename for journal files
;; :journal/file-name-format "yyyy_MM_dd"

399
pages/hls__ostep_1681115599584_0.md
View File

@ -1165,44 +1165,387 @@ file-path:: ../assets/ostep_1681115599584_0.pdf
ls-type:: annotation
id:: 64382ab8-ad41-4c20-be74-3ce7446f20d6
hl-color:: green
- Files And Directories
- ## Files And Directories
ls-type:: annotation
hl-page:: 498
hl-color:: yellow
id:: 6438d8bf-19fd-4a4a-b491-3887c425aebf
collapsed:: true
- File Directory
- File: a linear array of bytes, each of which can be read or written
- Directory: a list of *(user-readable name, low-level name)* pairs
- directory tree, root directory, separator, sub-directories, absolute pathname
- inode number: low-level name of a file or directory
- Creating Files: system call `open` with `O_CREAT` flag
hl-page:: 500
ls-type:: annotation
id:: 6438dd4b-55ba-4649-90fd-7de69ed9c2ba
hl-color:: yellow
- file descriptor: an integer, ==private per process==, used to access files
hl-page:: 501
- File operations
- Creating Files: system call `open` with `O_CREAT` flag
hl-page:: 500
ls-type:: annotation
id:: 6438dd78-68fe-47c6-8ca6-b2de3206f4f1
id:: 6438dd4b-55ba-4649-90fd-7de69ed9c2ba
hl-color:: yellow
- can be seen as a handle for file operations, or a pointer to an object of type file
- Reading And Writing Files: system call `read` and `write`
hl-page:: 502
ls-type:: annotation
id:: 6438e073-2636-4ec8-807d-3ecf83a5c0c0
hl-color:: yellow
- Non-sequential access: system call `lseek(fd, offset, whence)`
- explicit and implicit update to file offset
- open file table: represent all currently opened files in the system
- Shared File Table Entries
ls-type:: annotation
hl-page:: 506
hl-color:: yellow
id:: 6438e88b-e3e6-44a4-9f79-411c1f27ae71
- On every `open` call, the OS creates a new entry in the *open file table* even for the same file (same inode). Thus, they have independent offsets.
- Through `fork` or `dup`, we can make 2 file descriptors point to the same entry. In this case, reference count is needed to track when to release the entry.
- Figure 39.3: Processes Sharing An Open File Table Entry
- file descriptor: an integer, ==private per process==, used to access files
hl-page:: 501
ls-type:: annotation
hl-page:: 508
id:: 6438dd78-68fe-47c6-8ca6-b2de3206f4f1
hl-color:: yellow
id:: 6438e98b-b46f-4038-a984-eb172a628cc7
- can be seen as a handle for file operations, or a pointer to an object of type file
- Reading And Writing Files: system call `read` and `write`
hl-page:: 502
ls-type:: annotation
id:: 6438e073-2636-4ec8-807d-3ecf83a5c0c0
hl-color:: yellow
- Non-sequential access: system call `lseek(fd, offset, whence)`
- explicit and implicit update to file offset
- open file table: represent all currently opened files in the system
- Shared File Table Entries
ls-type:: annotation
hl-page:: 506
hl-color:: yellow
id:: 6438e88b-e3e6-44a4-9f79-411c1f27ae71
- On every `open` call, the OS creates a new entry in the *open file table* even for the same file (same inode). Thus, they have ==independent offsets==.
- Through `fork` or `dup`, we can make 2 file descriptors point to the ==same entry==. In this case, ==reference count== is needed to track when to release the entry.
- Figure 39.3: Processes Sharing An Open File Table Entry
ls-type:: annotation
hl-page:: 508
hl-color:: yellow
id:: 6438e98b-b46f-4038-a984-eb172a628cc7
- Writing Immediately
ls-type:: annotation
hl-page:: 509
hl-color:: yellow
id:: 643a7463-39e5-4645-8b43-b8be7b8ea7bc
- `write` system call is generally buffered, so the change may be applied to the disk some time later
- In rare cases, this could lead to data loss which is unacceptable for software like DB.j
- `fsync()` a particular file descriptor, and then the FS will force all dirty data to disk.
- In some cases, `fsync` the directory containing the target file is also necessary.
id:: 643a7594-e799-4dec-a9f6-fa94fada363f
- Renaming File: A special system call `rename(char*, char*)` for this, which is usually implemented to be atomic.
hl-page:: 509
ls-type:: annotation
id:: 643a782a-b874-4f34-9968-50b69a04b849
hl-color:: yellow
- File Information: `stat` or `fstat` system call which fetches information store in the file's *inode*
hl-page:: 510
ls-type:: annotation
id:: 643a789e-6346-4845-9d38-95d26927a32b
hl-color:: yellow
- Removing Files: `unlink`
hl-page:: 511
ls-type:: annotation
id:: 643a79be-d297-4f8d-8825-83c9f830af0e
hl-color:: yellow
- Directory Operations
- Making Directories: `mkdir`. Even an empty (newly created) directory has 2 entries: `.` and `..`
hl-page:: 512
ls-type:: annotation
id:: 643a7a52-0c85-435c-8250-3f4198a09fc0
hl-color:: yellow
- Reading Directories: 3 calls `opendir` `readdir` `closedir` and a `dirent` structure with a few fields. A directory "file" is comprised of many entries said above.
hl-page:: 513
ls-type:: annotation
id:: 643a7c32-0be4-4a2e-9532-31869f4e725a
hl-color:: yellow
- Deleting Directories: `rmdir`. Note that, this syscall can only remove empty directories or it will simply fail.
hl-page:: 514
ls-type:: annotation
id:: 643a7d55-768c-4881-b78d-f721c8d7929d
hl-color:: yellow
- Links
- Hard Links
hl-page:: 514
ls-type:: annotation
id:: 643a7d87-b945-4068-94dd-2b2d203eae67
hl-color:: yellow
- syscall `link` creates another name in the directory which refers to ==the same inode== of the original file.
- The *inode* keeps a reference count indicating how many hard links refer to it. On each `unlink`, RC decreases and the file will be deleted once the RC gets to 0.
- Hard links are essentially ==entries in directories== and hard links pointing to the same *inode* are just ==identical except their names==.
- An interesting usage of `link` is to rename, link to new and unlink the old
- Limitation: cannot link to a directory, cannot link to file on another partition (because *inode numbers* are only unique in the same FS/partition)
- Symbolic Links (Soft Links)
hl-page:: 516
ls-type:: annotation
id:: 643a8017-1499-4cd3-a015-0f1e8d143e93
hl-color:: yellow
- syscall `symlink`
- A symbolic link is essentially a ==special type of file==, which holds the pathname of the linked-to file.
- Dangling reference is possible, when then original file is deleted.
- Permission Bits
ls-type:: annotation
hl-page:: 518
hl-color:: yellow
id:: 643a84cb-102e-4beb-a138-e8690f68356f
- 10 characters (as shown in the out of `ls`)
- The left most indicates the type of the file, such as `-` for regular, `d` for directory, `s` for symbolic link and so on.
- The other characters are grouped by 3, each corresponding to a bit. Each 3-bit group indicates the permission of *owner*, *group* and *anyone*, and the bits means namely r/w/x. Note that as for directories, x bit represents the permission to enter directory.
- eponymous (与标题)同名的
hl-page:: 512
ls-type:: annotation
id:: 643a7a83-66d0-4bbc-8b17-bd7604a0ed5f
hl-color:: green
- hamster 仓鼠
hl-page:: 515
ls-type:: annotation
id:: 643a7dda-5c26-4ed5-8e06-c03e5a0e9fb7
hl-color:: green
- ## File System Implementation
hl-page:: 526
ls-type:: annotation
id:: 643a88bf-206b-4b6d-9d92-95516bcbe270
hl-color:: yellow
collapsed:: true
- The Mental Model: data structure of the FS and its access methods
hl-page:: 526
ls-type:: annotation
id:: 643a8c8e-b3ae-4de8-9280-75201b02d6db
hl-color:: yellow
- Disk Organization
- **blocks**: divide the disk into blocks (i.e. commonly used 4KB)
hl-page:: 527
ls-type:: annotation
id:: 643a8ce0-15b0-4d37-987d-6e649a59b616
hl-color:: yellow
- **data region**: most of the space is user data
hl-page:: 527
ls-type:: annotation
id:: 643a8ef8-7d3f-4794-9f00-4973a3be9bb7
hl-color:: yellow
- **inode table**: an array of inodes containing metadata to track per-file information
hl-page:: 528
ls-type:: annotation
id:: 643a8f4b-ac22-4cb8-b68d-e7f3078709c1
hl-color:: yellow
- **allocation structures**: recording information about free blocks, such as free list or bitmap. The `vsfs` from the book has one for inodes and another one for user data.
hl-page:: 528
ls-type:: annotation
id:: 643a8fa4-9783-4e86-8b04-2fad1f3a5c66
hl-color:: yellow
- **superblock**: information about the whole file system
hl-page:: 529
ls-type:: annotation
id:: 643a9022-da94-4092-8a64-e22cd2364365
hl-color:: yellow
- Inode
ls-type:: annotation
hl-page:: 529
hl-color:: yellow
id:: 643a92a1-90df-4342-a183-85c55242b8af
- **i-number**: each inode is referred to by a number (low-level name). Given an i-number, the on-disk location of the inode could be calculated fairly easily. Note that disk is not byte-addressed, need to read the whole sector.
- **metadata**: Inside each inode is virtually all of the information you need about a file. In addition, some necessary data for looking up data blocks.
- The Multi-Level Index
ls-type:: annotation
hl-page:: 531
hl-color:: yellow
id:: 643a93f0-e122-4ea5-a228-48607f93e404
- **direct pointer**: refer to one disk block that belongs to the file
- **indirect pointer**: points to a ==block that contains more pointers==, each pointing to a user data block.
- An inode may have s==ome fixed number of direct pointers==, and ==a single indirect pointer==. If a file grows large enough, an indirect block is allocated (from the ==data-block region of the disk==), and the inode’s slot for an indirect pointer is set to point to it. If even file grows even larger, add double/triple/... indirect pointer to your inode. The pointer refers to a ==block containing pointers to indirect blocks==, described above.
hl-page:: 531
ls-type:: annotation
id:: 643a950d-c2c2-419b-8206-0b957b7de178
hl-color:: yellow
- Figure 40.2: File System Measurement Summary -- Most files are small
hl-page:: 533
ls-type:: annotation
id:: 643a96d1-5114-40f7-9dad-4291462344ff
hl-color:: yellow
- Extent-based approaches
hl-page:: 532
ls-type:: annotation
id:: 643a975d-f0a2-47e7-bf85-63a300a2c504
hl-color:: yellow
- **extent**: a disk pointer and a length (how many contiguous blocks are there starting from the pointer)
- The advantage of this kind of approach is that, it is more compact, thus ==saving a lot of metadata==. The disadvantage is that, sometimes it is not easy to find many ==contiguous chunks==. Thus it works better when there is enough free space.
id:: 643a97ff-4b62-41bf-aa72-7316a8f3b974
- Linked-based approaches (FAT)
hl-page:: 534
ls-type:: annotation
id:: 643a9c2b-a519-4800-9188-253b13e9c2e5
hl-color:: yellow
- For each file, there is only one ==pointer to the first block== of the file. If more blocks are needed, add a ==pointer to another block at the end of this block==.
- Directory Organization
ls-type:: annotation
hl-page:: 533
hl-color:: yellow
id:: 643a990d-2b82-4bed-a49d-46746821b5a4
- A linear ==array of entries==. Each entry is a`(entry name, inode number)` pair, and perhaps an additional `length` (total bytes of this entry) and `strlen` (for the name).
- Directory, from the FS's perspective, is a ==special type of file==. Directories are allocated in *data region*, and also has corresponding inodes.
- Deleting a file can leave an empty space in the middle of the directory's block, and thus the FS needs to handle that (maybe mark it for reuse?).
- Free Space Management
ls-type:: annotation
hl-page:: 535
hl-color:: yellow
id:: 643a9d3c-3847-4a2e-b50e-f8423b089b39
- For our simple `vsfs`, search through the bitmap.
- pre-allocation: when allocating, look for a sequence of blocks (contiguous on disk) and give them to the new file, in order to improve performance
hl-page:: 535
ls-type:: annotation
id:: 643a9d6e-eab7-475c-9822-9b2efa21e6f0
hl-color:: yellow
- Reading A File From Disk
ls-type:: annotation
hl-page:: 536
hl-color:: yellow
id:: 643a9e6c-c2bd-48b9-80b8-b98340f91f7a
- Open: traverse the pathname and locate the desired inode. Since the inode of root directory `/` is fixed, we can start from `/`: load inode, read directory data, search for the next-level entry and recursively go down until the desired file's inode is loaded.
hl-page:: 536
ls-type:: annotation
id:: 643a9e8a-5f34-4dc9-817b-30466e84ffe2
hl-color:: yellow
- The amount of I/O generated by the open is proportional to the length of the pathname.
ls-type:: annotation
hl-page:: 537
hl-color:: yellow
id:: 643ab82b-d15c-4298-b52b-30c27da1deb0
- Read: first consult the file's inode for block address, and may update `access_time` field in the inode after read.
- Writing A File To Disk
ls-type:: annotation
hl-page:: 537
hl-color:: yellow
id:: 643ab83e-d2fe-4f02-b467-b969ab2482cc
- Write (with new block allocation): besides the cost of `open`, write to new block needs 5 IOs: read-write data bitmap (to allocate new free block), read-write inode (give the new block to the file), and finally write the block itself.
- Create: First of all, walk the path to its parent directory (a lot of IOs similar to `open`). Then, read-write inode bitmap (to allocate new inode block), write the new inode, read-write parent inode. Finally, if the directory block cannot accommodate the new file, more IOs generated.
- Caching and Buffering
ls-type:: annotation
hl-page:: 539
hl-color:: yellow
id:: 643abc13-2fdb-4a66-932a-eea1eafa449a
- static partitioning: a fixed-size cache to hold popular blocks with swap strategies such as LRU. Can be wasteful, though
hl-page:: 539
ls-type:: annotation
id:: 643abc6f-93a1-44e8-bb62-e56b2a9f2541
hl-color:: yellow
- dynamic partitioning: modern OSs integrate VM pages and FS pages into a unified page cache
hl-page:: 539
ls-type:: annotation
id:: 643abc73-d5b4-4627-9ab9-ef8863bc8f3d
hl-color:: yellow
- write buffering: FS can batch some updates into one IO to disk; FS can schedule subsequent IOs; some writes can be eliminated (such as an overwrite)
hl-page:: 540
ls-type:: annotation
id:: 643abc3c-040a-4f2f-89e4-ca14c9c0c230
hl-color:: yellow
- akin 相似的；类似的:
ls-type:: annotation
hl-page:: 532
hl-color:: green
id:: 643a96a0-f55d-4a84-bf93-4732a282b124
- readily 容易地；乐意地；
ls-type:: annotation
hl-page:: 533
hl-color:: green
id:: 643a96cb-bb7d-44c7-ac86-1e55a7ea3249
- per se 本质上，本身 by itself
hl-page:: 534
ls-type:: annotation
id:: 643a9be6-58e7-47a5-9ef2-bfff1f4c88a7
hl-color:: green
- bad mouth 说人坏话
hl-page:: 541
ls-type:: annotation
id:: 643abbf4-1d1a-4b32-9d91-345983406ce5
hl-color:: green
- ## Fast File System
ls-type:: annotation
hl-page:: 544
hl-color:: yellow
id:: 643abf69-04dc-428c-996d-b139cba0fa1f
collapsed:: true
- Problems with the rudimentary FS
- Data spread over the space, leading to long seek time between, e.g., inode and its data
- Free space get fragmented, making sequential read to a file slow
- Block size (512B) too small
- On-disk Structure
hl-page:: 546
ls-type:: annotation
id:: 643ac12f-817c-459b-b9ba-bbb1745519fb
hl-color:: yellow
- **cylinder**: ==a set of tracks== on different surfaces of a hard drive that are the ==same distance from the center== of the drive
hl-page:: 546
ls-type:: annotation
id:: 643ac141-3bff-4b00-9c18-57ad7c108fe5
hl-color:: yellow
- **cylinder groups**: aggregates consecutive cylinders into a group
hl-page:: 546
ls-type:: annotation
id:: 643ac144-24f3-46ca-9354-47c9d2f4ffe0
hl-color:: yellow
- **block groups**: modern HDDs tend not to expose internal structure, so the OS cannot have cylinder groups. Instead, organize the drive into block groups (they are consecutive anyways).
hl-page:: 547
ls-type:: annotation
id:: 643ac1ec-08d9-4584-b6ed-4332f8f791a0
hl-color:: yellow
- FFS spreads the components of a FS into each cylinder group.
- Each group has a copy of the super block.
- A per-group inode bitmap and data bitmap, keeping free block info in the group
- The remaining blocks are data blocks.
- Allocation Policies
hl-page:: 548
ls-type:: annotation
id:: 643ac42c-edf9-4b68-8890-b04d724c2ac2
hl-color:: yellow
- Basic mantra: Keep related stuff together.
- **Directory** placement: put the new directory in the cylinder group with a ==low number of allocated directories== (to balance directories across groups) and a ==high number of free inodes== (to subsequently be able to allocate a bunch of files)
hl-page:: 548
ls-type:: annotation
id:: 643ac55f-b4a2-4bfc-ad0b-23c785c4871e
hl-color:: yellow
- **File** placement: data blocks of a file in the ==same group as its inode== (long seeks between inode and data) , and it places files that are in the ==same directory== into the cylinder group of their enclosing directory.
hl-page:: 548
ls-type:: annotation
id:: 643ac616-ab90-40d5-94b2-7b8c294c4360
hl-color:: yellow
- The Large-File Exception
ls-type:: annotation
hl-page:: 551
hl-color:: yellow
id:: 643ac8cc-7164-4066-9730-231063c6c7c0
- Large file hurts locality by filling up the group and preventing other related files being placed in the same group
- After some number of blocks are allocated into the first block group, FFS places the next "l==arge" chunk of the file in another block group==. For example, put blocks pointed by direct pointers (12 blocks) in the first group, and those pointed to by the indirect block (1K blocks) in another.
hl-page:: 551
ls-type:: annotation
id:: 643ac9e7-1479-4be9-81fe-acb750f363b4
hl-color:: yellow
- Potential Performance: large sequential read from a large file. However, with selected chunk size (threshold of going to another group), ==cost of seek between groups can be amortized==. The larger size of a chunk, the higher average bandwidth you will reach.
- Measuring File Locality
hl-page:: 550
ls-type:: annotation
id:: 643ac769-6511-4954-a43a-957f3b561c56
hl-color:: yellow
- The metric can be: the distance to the common ancestor of the 2 files which are consecutively opened.
- About 40% are either the same or under same directory (FFS captures this), and 25% have distance of 2 (FFS failed to capture this)
- A Few Other Things About FFS
hl-page:: 553
ls-type:: annotation
id:: 643acef0-5592-4bad-8557-878567dc18a1
hl-color:: yellow
divide block into sub-blocks to save disk space (4K is too large for small files), and modify *libc* to buffer writes in 4KB chunks;
parameterization, a different disk block layout to improve sequential read
- replica 复制品
ls-type:: annotation
hl-page:: 547
hl-color:: green
id:: 643ac2c5-a62f-473b-b0e5-32807c7b2e6a
- corollary 必然的结果(或结论)
hl-page:: 548
ls-type:: annotation
id:: 643ac41b-c2e0-4351-b0e3-012e792cf426
hl-color:: green
- extensive 广阔的，广泛的；大量的，大规模的
ls-type:: annotation
hl-page:: 549
hl-color:: green
id:: 643ac4ec-df99-430a-9e2e-78101c74b14e
- nuance (意义、声音、颜色、感情等方面的)细微差别
hl-page:: 549
ls-type:: annotation
id:: 643ac4f0-6809-4acd-b141-2a96174f5395
hl-color:: green
- watershed 分水岭
hl-page:: 555
ls-type:: annotation
id:: 643acd1b-5a18-4b14-acdc-006c2f97c0e8
hl-color:: green
- ## Crash Consistency
hl-page:: 558
ls-type:: annotation
id:: 643acfc2-eef7-4c7f-a8a5-1740c8788159
hl-color:: yellow